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ABSTRACT 



Images sensed through object lenses 100R and 100L, having 
zoom lenses, with image sensors 102R and 102L are pro- 
cessed by image signal processors 104R and 104L, and an 
image of an object in each of the sensed images is separated 
from a background image on the basis of the processed 
image signals. The separated image signals representing the 
image of the object enter the image processor 220, where a 
three-dimensional shape of the object is extracted on the 
basis of parameters used upon sensing the images. The 
parameters are automatically adjusted so that images of the 
object fall within the both image sensing areas of the image 
sensors 102R and 102L and that they fall within the both 
focal depths of the image sensors 102R and 102L. 

30 Claims, 39 Drawing Sheets 
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IMAGE SENSING AND IMAGE PROCESSING partial scenes, and then is sensed with an electronic camera. 

APPARATUSES The images corresponding to the plurality of partial scenes 

is projected with having some overlapping portions. More 
specifically, the sensed images arc inputted into a personal 

BACKGROUND OF THE INVENTION 5 computer, then put together by using an application software 

The present invention relates to an image sensing appa- 50 that the overlapping portions are projected overlapping 

ratus capable of determining optimum image sensing con- each other - hereby, it is possible to obtain an image of far 

ditions for obtaining a three-dimensional image. The present Wlder an S le of view than 11131 of an ima S e obtained in a 

invention also relates to an image processing apparatus sm S le ima g e ^^ng operation by the electronic camera, 

which edits a three-dimensional image in accordance with 10 However, the main purpose of the aforesaid stereo imag- 

the image sensing conditions. ing method is to calculate information on distance between 

Conventionally, there is a technique, such as the one a fixed P ositioD where the camera is set and the object, and 

published on the Journal of Television Society, Vol. 45, No. n0 | to raeasure the three-dimensional shape of the entire 

4 (1991), pp. 453^60, to obtain a three-dimensional shape object. 

of an object. There are basically two methods, a passive Further, since a laser beam is emitted to the object in the 

method and an active method, for obtaining the three- active method, it is troublesome to use an apparatus adopting 

dimensional shape of the object, as described in the above the active method. Furthermore, in any conventional 

article. method, cameras are not controlled flexibly enough to 

A typical method as the passive method is a stereo 20 respond to a dynamic image sensing method, such as sensing 

imaging method which performs triangulation on an object an object while moving around it. 

by using two cameras. In this method, corresponding points In addition, two view finders are necessary in the con- 
of a part of an object are searched in both the right and left ventional passive method using two cameras, and it is also 
images, and the position of the object in the three dimen- necessary to perform image sensing operation as seeing to 
sional space is measured on the basis of the difference 1S compare images on the two view finders, which increases 
between the positions of the searched corresponding points manufacturing cost and provides bad operability. For 
in the right and left images. instance, there are problems in which it takes time to 
Further, there are a method using a range finder and a slit Perform framing or it becomes impossible to obtain a 
projection method as the typical active method. In the three-dimensional shape because of too small of an over- 
former method, distance to the object is obtained by mea- 30 * a PP in S area * 

suring the elapsed time between emitting light toward the Further, an image generally dealt with in an office is often 

object and receiving the light reflected by the object. In the printed out on paper eventually, and types of images to be 

latter method, a three-dimensional shape is measured on the used may be a natural image and a wire image which 

basis of deformation of a shape of a light pattern, whose represents an object with outlines only. In the conventional 

original pattern is a slit shape, projected on an object. 35 methods, however, to display an image of an object faith- 

However, the main purpose of the aforesaid stereo imag- 00 a two-dimensional display on the basis of three- 

ing method is to calculate information on distance between dimensional shape data of the object is the main interest, 

fixed positions where the cameras are set and the object, and tDUS those methods are not used in ofiSces. 

not to measure the entire object. Consequently, a three- cnmnBvnriTJD ixnn:vrnAM 

dimensional shape can not be obtained in high precision. 40 SUMMARY OF THE INVENTION 

Further, an apparatus adopting the active method is large, The present invention has been made in consideration of 

since it has to emit a laser beam, for example, to an object the aforesaid situation, and has as its object to provide an 

the manufacturing cost is, therefore, high, image sensing apparatus capable of placing an object, whose 

Generation of a three-dimensional shape of the object on three-dimensional shape is to be generated, under the opti- 

the basis of two-dimensional images requires a plurality of 45 mum ima g e sensing conditions upon sensing the object from 

images sensed at a plurality of image sensing points. a plurality of image sensing points without bothering an 

However, since the object has a three-dimensional shape, operator. 

image sensing parameters (e.g., depth of focus, angle of It is another object of the present invention to provide an 

view) which are suited to the object and each image sensing image sensing apparatus capable of setting sensing param- 

point needs to be set for performing image sensing from a 50 eters for an optical system so that an entire object falls 

plurality of image sensing points. within the optimum depth of focus at image sensing points. 

However, in any of the aforesaid methods, cameras are A further object of the present invention is to provide an 

not controlled flexibly enough to respond to a dynamic image sensing apparatus which senses an image of the object 

image sensing method, such as sensing an object while 55 with the optimum zoom ratio at each of a plurality of image 

moving around it. sensing points. 

Therefore, the present invention is aimed at solving the Yet a further object of the present invention is to provide 

aforesaid problem, i.e., to realize a dynamic image sensing an image sensing apparatus capable of notifying an operator 

method in which an object is sensed at a plurality of image of achievement of the optimum image sensing conditions, 

sensing points around it. 60 Yet further object of the present invention is to provide an 

Meanwhile, an image of an object which is seen from an image sensing apparatus capable of storing the optimum 

arbitrary viewpoint is sometimes reproduced on a two- image sensing conditions. 

dimensional display on the basis of obtained three- Yet a further object of the present invention is to provide 

dimensional data of the object. an image sensing apparatus which determines whether the 

For example, it is possible to input images sensed by an 65 optimum image sensing conditions are achieved or not by 

electronic camera into a personal computer, or the like, and judging whether there is a predetermined pattern in an image 

edit them. In this case, a scene is divided into a plurality of sensing field. 
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Yet a further object of the present invention is to provide 
an image sensing apparatus capable of re-sensing an image. 

Yet a further object of the present invention is to provide 
an image sensing apparatus whose operability is greatly 
improved by informing an operator when he/she is to press 
a shutter. 

Yet a further object of the present invention is to provide 
an image sensing apparatus capable of determining a dis- 
placing speed of a camera upon inputting an image, thereby 
improving operability as well as quality of an input image. 

Yet a further object of the present invention is to provide 
a single-eye type image sensing apparatus capable of input- 
ting an image of high quality thereby obtaining a three- 
dimensional shape in high precision and reliability. 

Yet a further object of the present invention is to provide 
an image sensing apparatus capable of always sensing 
characteristic points to be used for posture detection within 
a field of view, thereby preventing failing an image sensing 
operation. 

Yet a further object of the present invention is to provide 
an image processing apparatus capable of generating an 
image of an object which is seen from an arbitrary viewpoint 
on the basis of three-dimensional shape information on 
images sensed at a plurality of image sensing points, and 
capable of forming a file. 

Yet a further object of the present invention is to provide 
an image processing apparatus capable of generating an 
image of an object which is seen from an arbitrary viewpoint 
on the basis of three-dimensional shape information on 
images sensed at a plurality of image sensing points, and 
capable of forming a file, and further editing the three- 
dimensional image by synthesizing the file with other file. 

Yet a further object of the present invention is to provide 
an image processing apparatus which generates a three- 
dimensional image from images sensed under the optimum 
image sensing conditions. 

Yet a further object of the present invention is to provide 
an image processing apparatus which converts three- 
dimensional shape data, obtained based on sensed images, 
into a two-dimensional image of an object which is seen 
from an arbitrary viewpoint. 

Yet a further object of the present invention is to provide 
an image processing apparatus which combines a document 
file and a three-dimensional image. 

Yet a further object of the present invention is to provide 
an image processing apparatus which stores information on 
background of an object. 

Yet a further object of the present invention is to provide 
an image processing apparatus which combines an image 
data file with another file, and has a three-dimensionally 
displaying function. 

Yet a further object of the present invention is to provide 
an image processing apparatus in which three-dimensional 
shape data of an object is calculated with a software installed 
in a computer, the three-dimensional shape data of the object 
is converted into an image of the object seen from an 
arbitrary viewpoint, and a file of the image data is combined 
with another file. 

Yet a further object of the present invention is to provide 
an image sensing apparatus capable of detecting overlapping 
areas in a plurality of images sensed at a plurality of image 
sensing points. 

Yet a further object of the present invention is to provide 
an image sensing apparatus which displays overlapping 
portions of images sensed at a plurality of image sensing 
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points in a style different from a style for displaying non- 
overlapping portions. 

Yet a further object of the present invention is to provide 
an image processing apparatus capable of re-sensing an 
5 image. 

Other features and advantages of the present invention 
will be apparent from the following description taken in 
conjunction with the accompanying drawings, in which like 
reference characters designate the same or similar parts 
10 throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated in 
and constitute a part of the specification, illustrate embodi- 
15 ments of the invention and, together with the description, 
serve to explain the principles of the invention. 

FIG. 1 is an explanatory view for explaining modes set in 
the image processing apparatuses according to first and 
second embodiments; 

20 

FIG. 2 is an overall view illustrating a configuration of a 
three-dimensional shape recognition apparatus according to 
a first embodiment of the present invention; 

FIG. 3 is a block diagram illustrating a configuration of a 
25 three dimensional shape information extracting apparatus 
according to the first embodiment; 

FIG. 4 is a block diagram illustrating a configuration of a 
system controller shown in FIG. 3; 

FIGS. 5 A, 5B are a flowchart showing an operation 
30 according to the first embodiment; 

FIG. 6 is an explanatory view for explaining in-focus 
point adjustment; 

FIGS. 7A to 7D are explanatory views showing zoom 
ratio adjustment; 

FIG. 8 is a graph for explaining the zoom ratio adjust- 
ment; 

FIG. 9 is an overall view illustrating a brief configuration 
of a three-dimensional shape recognition apparatus accord- 
40 ing to a first modification of the first embodiment; 

FIG. 10 is a block diagram illustrating a configuration of 
a three dimensional shape information extracting apparatus 
according to the first modification; 

FIG. 11 is a flowchart showing an operation according to 
45 the first modification; 

FIG. 12 is an explanatory view showing a principle of 
detecting a posture according to the first modification; 

FIG. 13 is a block diagram illustrating a configuration of 
a three dimensional shape information extracting apparatus 
50 of a second modification of the first embodiment; 

FIG. 14 is a flowchart showing an operation according to 
the second modification; 

FIG. 15 is a brief view of a three-dimensional shape 
ss extraction apparatus and its peripheral equipment according 
to the second embodiment; 

FIG. 16 shows types of images of an object according to 
the second embodiment; 

FIGS. 17A, 17B are a block diagram illustrating a detailed 
60 configuration of an image sensing head and an image 
processing unit; 

FIG. 18 is a block diagram illustrating a configuration of 
a system controller; 
FIG. 19 shows images of an object seen from variety of 
65 viewpoints; 

FIG. 20 is a diagram showing a flow of control by the 
apparatus according to the second embodiment; 
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FIG. 21 is an explanatory view for explaining a principle plurality of image sensing points, generates a three- 

of detecting an overlapping portion according to the second dimensional image from these images and displays it. An 

embodiment; image processing apparatus described in a first embodiment 

FIG. 22 is an example of an image displayed on a finder is characterized by three-dimensional shape recognition, 

according to the second embodiment; 5 especially wherein the optimum image sensing parameters 

FIG. 23 is an table showing a form of recording three- are decided u P°n to sense the images of the object. As for an 

dimensional images on a recorder according to the second ima S c processing apparatus described in a second 

embodiment* embodiment, it is characterized by correcting a three- 

HG. 24 shows an operation in a panoramic image sensing dilnensi ° Qal on the basis of predetermined image 

according to the second embodiment; 10 ""I P aramete « ° r edlto g * three-chmensional image. 

FIG. 25 shows an operation in a panoramic image sensing FIG. 1 is an explanatory view for explaining modes set in 

according to the second embodiment; the P">«*^g apparatuses according to the first and 

Z, . , . r n . tl . - . second embodiments. In an image sensing parameter deter- 

FIG. 26 shows a brief flow for calculating d.stance mo(J ^ ifflum ; ^ meters are 

information from stereo images according to the second 1S determined In a t ,f r ee -dimensional shape information 

em o imen , extraction mode, three-dimensional shape information of an 

FIG. 27 is an explanatory view for explaining a principle ob j ect ^ extracted from images of the object sensed at a 

of a template matching according to the second embodi- plurality of image sensing points by using image sensing 

menl » parameters determined in the image sensing parameter 

FIG. 28 shows a brief flow for combining the distance 20 determination mode. In a display and editing mode, a 

information according to the second embodiment; three-dimensional image is configured from three- 

HG. 29 is an explanatory view for briefly explaining an dimensional shape information, displayed and further 

interpolation method according to the second embodiment; edited. In a panoramic image sensing mode, a panoramic 

FIGS. 30A and 30B are explanatory view for showing a image is generated by synthesizing images sensed at a 

method of mapping the distance information to integrated 25 plurality of image sensing points by using a function, 

coordinate systems according to the second embodiment; originally performed for extracting three-dimensional shape 

FIG. 31 shows a brief coordinate system of an image information, for sensing a plurality of images from a plu- 

sensing system according to the second embodiment; ralil y °. f ima S e sensm S P oints - Further > a three-dimensional 

FIG. 32 shows a brief coordinate system when the image 30 ima S e m P ut mode is rmshed. 

sensing system is rotated according to the second embodi- Embodiment 
ment; 

FIG. 33 shows a brief flow of combining a document file, ADetermination of Image Sensing Parameters 

image information arid the distance information according to ,~ „ ~ ~ .. 

. 6 , , & Overall Configuration 

the second embodiment; 35 

FIG. 34 is an explanatory view showing a flow that image FIG. 2 is an overall view illustrating a configuration of a 

information is fitted to a model image according to the three-dimensional shape recognition apparatus according to 

second embodiment' a ^ rst embodiment of the present invention. It is necessary 

FIG. 35 is an explanatory view showing that an image t0 decide the most suitable image sensing parameters for 

information file is combined with the document file accord- 40 recognizing three-dimensional shape information from 

ing to the second embodiment; images. The most suitable image sensing parameters make it 

-i*- • u • r « • r • easy to recognize, in high precision, three-dimensional 

FIG. 36 is a brief overall view of an image processing , J . c * c , . /„, t , . , . 

j . , « . ,. c c *l j shape information or an object. The three-dimensional shape 

system according to a first modification oi the second •* • . L • i j « *u j r 

J , .. & recognition apparatus shown in FIG. 2 adopts a method ot 

embodiment; j . • • • • * c.u 

45 determining image sensing parameters or the present mven- 

F1G. 37 is a block diagram illustrating a configuration of tion ^ order t0 realize reHable three-dimensional shape 

a three-dimensional shape extraction apparatus 2100 accord- reC ognition in high precision. More specifically, in the 

ing to the first modification of the second embodiment; apparatus shown in FIG. 2, after the most suitable image 

FIG. 38 is a flowchart showing a processing by the sensing parameters are determined, an optical system is set 

three-dimensional shape extraction apparatus according to 5Q m accordance with the image sensing parameters, then the 

the first modification of the second embodiment; apparatus senses images of the object and recognizes three- 

FIG. 39 is a block diagram illustrating a configuration of dimensional shape information of the object of interest, 

a three-dimensional shape extraction apparatus according to \ n FIG. 2, reference numeral 1 denotes an apparatus 

the second modification of the second embodiment; (called "three-dimensional shape information extracting 

FIG. 40 is a flowchart showing a processing by the 55 apparatus", hereinafter) for extracting three-dimensional 

three-dimensional shape extraction apparatus according to shape information of an object; and 2, an object whose 

the second modification of the second embodiment; and three-dimensional shape information is to be extracted, and 

FIG. 41 is an example of an image seen on a finder the object 2 becomes a subject of a camera for obtaining the 

according to a third modification of the second embodiment. three-dimensional shape information of the object by an 

nn-TAiT en nccPDiDTinM nv ttjc 60 image P rocessin S in tne P resent invention. Further, reference 

DhlAILbD DESCRI1 HON OF IHh numeral 3 denotes a stage set behind the object 2, and it 

PREFERRED EMBODIMENTS constitutes a background of the object 2. 

Preferred embodiments of the present invention will be In the three dimensional shape information extracting 

described in detail in accordance with the accompanying apparatus 1, reference numeral 100R denotes a right object 

drawings. 65 lens; and 100L, a left object lens. Further, reference numeral 

The present invention discloses an image processing 200 denotes an illumination unit for illuminating the object 

apparatus which obtains images of an object sensed at a 2 in accordance with an image sensing environment. The 
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image sensing field of the right object lens 100R is denoted Images of the object 2 are inputted to the image sensors 

by reference numeral 10R, and the image sensing field of the 102R and 102L through the object lenses 100R and 100L, 

left object lens 100L is denoted by reference numeral 10L. and converted into electrical image signals. The obtained 

The three dimensional shape information extracting appa- electrical image signals are converted from analog signals to 

ratus 1 is mounted on a vibration-type gyro (not shown), for 5 digital signals by the A/D converters 1G3R and 103L and 

example, and the position of the three dimensional shape supplied to the image signal processors 104R and 104L. 

information extracting apparatus 1 is detected by a posture The image signal processors 104R and 104L convert the 

detector 201 (refer to FIG. 3) which is also mounted on the digitized image signals of the object into luminance signals 

vibration-type gyro. anc j color signals (image signals in the YIQ system or image 

The three dimensional shape information extracting appa- i° signals in the Lab system as described above) of a proper 

ratus 1 senses the object 2 while moving from the start format. 

position Aq of the image sensing to the end position A„ of the Nexl> the image separalors 105R and 105L separates the 

image sensing. Further, position information and posture image of lne ob j ect whose three-dimensional shape infor- 

information of the three dimensional shape information mation is the subject to measurement from the image of the 

extracting apparatus 1 at each image sensing point between 35 background 3 i n trie sensed image signals on the basis of the 

Aq-A„ are calculated from signals obtained from the posture signals obtained from the image signal processors 104R and 

detector 201 of FIG. 3. 104L . 

FIG. 3 is a block diagram illustrating a configuration of M aD example of a separation method, first, sense an 

the three dimensional shape information extracting appara- of the background in advance and store the sensed 

tus (referred to as "parameter extracting apparatus" ^ the memory (F ] Gi 4 y Then, place the object 2 to 

hereinafter) 1. be measured in front of the background 3 and sense an image 

In FIG. 3, reference numerals 100R and 100L denote the 0 f the object 2. Thereafter, perform matching process and a 

object lenses consisting of zoom lenses. Further, reference differentiation process on the sensed image including the 

numerals 101R and 101L denote iris diaphragms; and 102R ^ object 2 and the background 3 and the image of the back- 

and 102L, image sensors and CCDs can be used as those. ground 3 which has been stored in the memory in advance, 

A/D converters 103R and 103L convert signals from the thereby separating the areas of the background 3. It should 

image sensors into digital signals. Image signal processors be noted that the separation method is not limited to the 

104R and 104L convert the digital signals from the A/D above, and it is possible to separate images on the basis of 

converters 103R and 103L into image signals of a predeter- ^ q information on colors or texture in the image, 

mined format (e.g., image signals in the YIQ system or Xhe ^urted image signals of the object 2 are inputted 

image signals in the Lab system). Image separators 105R t0 the processor 2 20, where three-dimensional shape 

and 105L separate an image of the object 2 from an image exlract ion is performed on the basis of the image sensing 

of the background 3. parameters at the image sensing operation. 

Zoom controllers 106R and 106L adjust focal lengths of 35 Next> an operational seq uence of the system controller 

the object (zoom) lenses 100R and 100L. Focus controllers no of ^ ameler extracting apparatus 1 will be 

107R and 107L adjust focal points. Iris diaphragm control- described with re ference to a flowchart shown in FIGS. 5A, 
lers 108R and 108L adjust aperture diaphragm or the ins 

diaphragms 101R and 101L. , , r. . . • « * « n r 

J[ r 1 ™» j . *i_ 1 In the flowchart shown in FIGS. 5 A, 5B, processes from 

Reference numeral 201 denotes the posture detector 4n , n C1 . CQ , . . «. „ 0 „ OMmot „ 

... • . j j ■ . step SI to step S9 relate to the 'image sensing parameter 

which consists of a vibration-type gyro and so on, and it , . , „ , „. * _ • „ „, M ™«t«- 

. . ,. iL Jr , c .1 determination mode . In the image sensing parameter 

outputs signals indicating the position and posture of the , t . . , „ . r °. r 

v n b c 1 11 n j * . * n determination mode , the optimum image sensing param- 

camera. Reference numeral 210 denotes a system controller a . , an , - n ■ ' . „ * A . ? V,, 

... . , „ . eters for each of n image sensing points, or Ao to A„ shown 

which controls the overall parameter extracting apparatus 1. in FIG 2 are determined 

The system controller 210, as shown in FIG. 4, consists of 45 , , 

a microcomputer 900, memory 910 and an image processing When a P ower switch 15 mraed on > each unit shown m 

section 920. Reference numeral 220 denotes an image FIG. 3 starts operating. When the "image sensing parameter 

processor which extracts three-dimensional information of determination mode" is selected, the system controller 210 

the object on the basis of the image signals obtained from the starts controlling at step SI. More specifically, the system 

image sensors 102R and 102L, as well as outputs data after 50 controller 210 enables the iris diaphragm controllers 108R 

combining the three-dimensional information extracted at and 108L and the ima S e seQSOrs 102R and 102L 50 as t0 

each image sensing point and posture information at each make them 0Ut P ut ima 8 e S1 8 nals throu S h the lenses 

image sensing point obtained by the posture detector 201. 100R and 100L » enables the converters 103R and 103L 

Reference numeral 250 denotes a recorder for recording an 50 38 t0 make them converl the ima g e sl S nals into dl g> lal 

^ ma g e image signals, and controls the image signal processors 

A focusing state detector 270 detects a focusing state of 104R , and 104L t0 mak ^ them convert the digital image 

the sensed image on the basis of the image of the object 2 signals into image signals of the aforesaid predetermined 

and the image of the background 3 separated by the image format ( includes *™ ina °ce component, at least), 

separators 105R and 105L. An R-L difference discriminator As the image signal processors 104R and 104L start 

260 calculates the differences between the obtained right 60 outputting the image signals, the system controller 210 

image sensing parameters and left image sensing param- adjusts exposure at step S2. 
eters. 

Furthermore, reference numeral 230 denotes a shutter; ^ 
280, an external interface for external input; and 240, a The system controller 210 controls the image processing 
display, such as a LED. 65 section 920 (refer to FIG. 4) to perform an integral process- 
Next, an operation of the parameter extracting apparatus ing on the image signals of the object 2 obtained from the 
1 having the aforesaid configuration will be explained. image separators 105R and 105L, and calculates a lumi- 
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nance level of the image of the entire object 2. Further, the Alternatively, proper weights may be applied, and the equa- 

system controller 210 controls the iris diaphragm controllers tion (1) becomes, 
108R and 108L to set the iris diaphragms 101R and 101L to 

proper aperture diaphragms on the basis of the luminance x _ mX v +n x s (4) 

level. At step S3, whether the luminance level obtained at 5 ~ 2 
step S2 is not high enough to extract three-dimensional 

shape information and any control of the iris diaphragms The range in the X direction in which the entire object 2 

101R and 101L will not result in obtaining a proper lumi- is within the depth of focus of the lenses 100R and 100L is 

nance level or not is determined. If it is determined that the denoted by X^-X^ when the object lenses 100R and 100L 

proper level is not obtained, then the illumination unit 200 10 focus at the distance X denoted by the equation (1). The 

is turned on at step S4. Note, the intensity level of the upper limit of the range X/ and the lower limit of the range 

illumination unit 200 may be changed on the basis of the X 2 ' can be expressed as follows, 
luminance level calculated at step S2. 

is , Y ; (5) 

In-focus Point Adjustment (Step S5) ' f l + & ' F ' - /> 

The system controller 210 adjusts the focal lengths at step x$ = ri+* 2 f\x ^ 

S5 by using the right and left image signals which are set to 2 

a proper luminance level. The parameter extracting appara- 20 

tus 1 shown in FIG. 3 has the focus controllers 107R and Here, f denotes the focal length of the lenses 100R and 

107L, thus adjustment for focusing is unnecessary. 100L, F denotes a F number (^aperture diameter/focal 

Therefore, the in-focus point adjustment performed at step length), and 5 denotes the diameter of the circle of least 

S5 is an adjustment of focus so that the entire image of the confusion. It should be noted that the size of a pixel of the 

object is within the focal depths of the lenses 100R and 25 image sensors 102R and 102L can be considered as 5, for 

100L. example. 

Accordingly, in a case where the system controller 210 

A principle of the in-focus point adjustment process tries to control the lenses 100R and 100L to focus at the 

performed at step S5 will be shown in FIG. 6. distance X expressed by the equation (1), it can obtain a 

^. , # , . f . . e4U , inniJ j mm 30 clear image of the object 2 in the aforesaid range between 

First, the in -focus points of the lenses 100R and 100L are v . v , S. c . * n * 1 * L 

, ' * 1 > XZ-XV. Therefore, the system controller 210 controls the 

setattheu PP er P artoftheob J ect2 then set at the lower part ^ d - h or me e diameter of the iris 

of the object 2. The lower part of the object 2 can not be £ h ^ b 6 y mnUo \\in g fhe iris diaphragm controllers 

usually seen from the lenses 100R and 100L, therefore the 1Q8R and 108L so that the p number which achieves the 

in-focus points of the lenses 100R and 100L are adjusted to 35 dosest match between respective X ; and satisfying the 

the background 3 in practice at step S5. equations (5) and (6), and X, and X 2 of the equations (1) to 

Note, the focusing state in this process is detected by the (3). 

focusing state detector 270. As for a detection method, a Thus, the focal length and aperture diaphragm are deter- 

known method, such as detection of clarity of edges or mined so that a clear image can be obtained in the entire 

detection of a blur from image signals, may be used. 40 range in the depth direction of the object 2 (in the X 

. r . . direction) by performing the operational sequence of steps 

The aforesaid focusing operation on the two parts, i.e., the g2 to S5 

upper and the lower parts of the object, are performed for ^ m fl case wfaere the luminance levd is changed more 

each of the right and left lenses 100R and 100L, reference thaQ a predetermined value 5y the proce ssing at step S5, it 

numerals X, and X 2 m FIG. 6 represent focus lengths to the 45 cafl ^ ^ wim by changing ^ intensity of the illumina . 

upper and lower parts of the object 2 of enher the right or left ^ m{i 2m {Q dea , whh this simtioa ^ t0 

tens. The focusing state detector 270 outputs the values of additiooally provide an AGC (automatic gain control) circuit 

Xl M ?, \t SyStCm 21 ?' ™en the system to correct the level electrically, 
controller 210 determines a focal length X with which the 

depth of focus is determined for the object 2 on the basis of 5Q Zoom Ratio Adjustment (Step S6) 

the values, then outputs a control signal so as to obtain the Next at step S6j ratio ^ adjusted so that the entire 

focus distance X to the corresponding focus controller 107R ob j ect 2 [ s [ n the field of view of the camera. In order to 

or 107L. The distance X may be a middle length between X, generate a three-dimensional image of the object, there has 

and X 2 , for example, t0 be an overlapping portion in images sensed at at least two 

55 image sensing points. In a case where the convergence angle 

X = X{ +Xz between the right and left lenses is much different from the 

2 angles of view of the right and left lenses, there would not 

be any overlapping portion in the images. Therefore, by 

. t . r , f( , . . j i c. 1 maximizing an overlapping portion, a three-dimensional 

In practice, for each of the right and left lenses, c 0 u v -1 

r & image of a wide area can be realized. 



x 1R + x 2R (2) 



60 

FIGS. 7A, 7B, 7C and 7D are explanatory views showing 
Xr - 2 " iK v ~ / a brief zoom ratio adjustment performed at step S6. 

The system controller 210 stores images obtained from 
x L = XlL + XlL (3) the image sensors 102R and 102L when the object 2 is 

2 65 basically in the focal depth X 1 '-X 2 ' in the memory 910 (FIG. 

4) as well as detects an overlapping portion of the object 2 
by using the image processing section 920. The overlapping 
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portion is represented by image signals included in both in the form of parameters which specify the shape of the 

images of the object sensed by the right and left lenses. The object. The aforesaid X : and X 2 are only the distances in the 

overlapping portion is shown with an oblique stripe pattern camera space coordinate system, not the real distances to the 

in FIGS. 7A, 7B, 7C and 7D. As a method of detecting the object. Therefore, the parameter extracting apparatus of the 

overlapping portion, a correlation operating method which 5 first embodiment finds the actual distance to the object as an 

takes correlation by comparing the obtained right and left example of shape parameters. 

^^^Z^^&^^S lh ™T 6 infora f on z lo the ob * ct can be ex " rcsse d * 

and left images, for instance, may be used, the followuig equation. 

The zoom ratio adjustment at step S6 is for adjusting the 3Q 
zoom ratio so that the area of the overlapping portion z = — (7J 

becomes maximum. d 

In order to do so, the controller 210 detects the overlap- 
ping portion 500 between the right and left images of the Here, f denotes a focal length of the optical system; b, base 

object 2 sensed by the right and left lenses 100R and 100L, ^ line length; and d, parallax. 

as shown in FIGS. 7A and 7B, by using the aforesaid 1 i n order to recognize three-dimensional shape informa- 
method. Next, the zoom ratios of the lenses 100R and 100L t i on { n better precision by performing image processing, the 
are changed so as to increase the area of the overlapping resolving power at the distance Z with respect to the parallax 
portion 500 (e.g., in FIGS. 7C and 7D), then the controller ^ important. The resolving power at the distance Z is defined 

210 outputs control signals to the zoom controllers 106R and ^ by the following equation. 

FIG. 8 is a graph showing change of the area of the $ z f-b (8) 

overlapping portion 500 of the object 2 in frames in accor- Jd = ~~d r 

dance with the zoom ratio adjustment. 

In FIG. 8, the focal length f of the lenses 100R and 100L « f ■ « .u , - • . . 

* u- u *u i • f M u .l 25 In this system, the resolving power is considered as one 

at which the area of the overlapping portion 500 reaches the r * u • • . j * * j * u 

peak P is calculated by the image processing section 920 of of , th / TJ" 8 parame 'f ™ d const i ructe . d fl . so a f. t ° be 

The controller 210, then the controller 210 gives a control ou * lde , an °P erator - ^ . e 1 uaU ° D f '? d / ca * s 

signal to the zoom controllers 106R and 106L so as to obtain that ' when ^ re f lvm 8 P™* 1 18 S' ven * the foca lel f h 

the focal len th f changes in accordance with the resolving power. In other 

' 30 words, 
Accordingly, by determining the optimum exposure 

condition, the optimum aperture diaphragm and the opti- 2 (g 

mum focal length f at steps S2 to S6, it is possible to obtain / - _ ^_ . 

a clear image of the entire object 2 both in the depth and 
width directions. 35 

Readjustment of Parameters ,l * Possible to set the resolving power from an external 

computer, or the like, through the external I/F 280 at step S9 

In a case where the focal length f is changed by the and to set a focal length f in accordance with the equation (9) 

operation at step S6 which results in changing the depth of based on the resolving power upon adjusting focal length at 

focus more than a predetermined value (YES at step S7), the 4Q step S5 m the flowchart shown in FIGS. 5A, 5B. 

process proceeds to step S8 where the parameters are Accordingly, the optimum image sensing parameters for 

readjusted. The readjustment performed at step S8 is to each of n image sensing points, Aq to A,,, can be determined 

repeat the processes at steps S2 to S7. from the proce sses at steps S2 to S9. 

Adjustment of Right-Left Difference Shape Recogaition 

In the adjustment of right-left difference performed at step « , n ^ sub m st after st S10 an { of the 

S8, the right-left differences of the exposure amounts obje ct 2 is sensed. The image sensmg is for recognizing the 

(aperture diaphragm), the aperture values b and the zoom sh of ^ ^ - aQd ^ . 

must be sensed under 

ratios (focal length f) of the right .and left optical systems are ^ ^ condiljon for , he Uiice^limci«fonal 

detected by the R-L difference discriminator 260, and each sh recogmtion by ^ the image xnsing pinmtlers 

optical system is controlled so that these differences 50 detemjned a , S2 t0 s , 

decrease. More specifically, in a case where differences r . , . „_ . , 

between the right and left lenses on the aperture diaphragms fXf em controller 210 gives a signal to the 

(will affect the exposure amounts and the focal depths) and display 240 at step S10, and notifies a user of the completion 

the focal lengths (will affect the focal depths and an angles ° f *V™ ge S Parameters. The display 240 can be a 

of field of view) obtained for the right and left optical 55 CRT, LCD, oi :thc ^ like, or can be a simple dasplay using an 

systems at steps SI to S7 are more than a threshold, the ^ED, or the like. Fur her, sound may be used for notification 

image sensing parameters obtained for the right optical aloD § Wlth thc visual dls P la y- 

system are used for both of the right and left optical systems. Next > the user confirms the notification by the LED, or the 

In other words, if the respective image sensing parameters ^ "J** 8 the s y stem controller 210 start performing 

for the right optical system differ from the corresponding 60 the three-dimensional recognition (extraction of shape 

image sensing parameters for the left optical system, then parameters). 

the image sensing parameters for the right optical system is Wnen the user pushes a start inputting button (not shown) 

used as the parameters for the system shown in FIG. 2. at ste P Sll, the posture of the camera is initialized at step 

S12. The initialization of the posture of the camera per- 

Setting of Resolving Power (Step S9) 65 formed at slep su & to initialize the posture of the camera 

In recognizing three-dimensional shape information, by using the optimum image sensing parameters for an 

information for expressing the actual distance to an object is image sensing point of the camera, A,,., obtained at steps S2 
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to S9, where the camera currently is. This initialization 
guarantees that the entire object falls within a proper depths 
of focus. 

Next, the parameter extracting apparatus 1 senses the 
object 2 from different positions while moving from the 5 
position Aq toward the position A„ as shown in FIG. 2. In 
this case, the change of the image sensing parameters is 
prohibited by the system controller 210 since the start 
inputting button is pushed until a stop button is pushed. 

While the parameter extracting apparatus 1 moves, the 10 
posture and the speed of the displacement of the apparatus 
1 are detected by the posture detector 201 provided inside of 
the apparatus 1 at step S13. In a case where the detected 
speed of the displacement is not within an appropriate range 
at step S14, the user is notified by lighting the LED of the 15 
display 240 at step S15. 

At step SI 6, the user inputs an image by pressing the 
shutter 230 at a predetermined rate. It is possible to notify 
the user of the timing to press the shutter 230 calculated by 2Q 
the system controller 210 on the basis of the signals detected 
by the posture detector 201, by lighting the LED of the 
display 240, for example. 

When it is detected that the shutter 230 is pressed, in 
synchronization with the detection, the system controller 2 s 
210 calculates posture information at that point from the 
signals detected by the posture detector 201 at step SI 7, as 
well as gives the calculated posture information to the image 
processor 220. Thereafter, at step S18, the image processor 
220 calculates the three dimensional coordinates of the 30 
objects from the posture information and the image signals 
of the right and left images, then outputs the calculated 
coordinates to the recorder 250 along with pixel values 
thereof. 

At step S19, the recorder 250 converts the inputted signals 35 
to signals in a proper format, and writes the formatted 
signals on a recording medium. After that, the processes at 
step S17 to S20 are repeated until it is determined that all the 
image signals have been inputted at step S20. When all the 
image signals have been inputted, the process is completed 40 
at step S21. 

First Modification 

FIG. 9 is an overall view illustrating a brief configuration 
of a three-dimensional shape recognition apparatus accord- 45 
ing to a modification of the first embodiment. In contrast to 
a double-eye camera used in the first embodiment, a single - 
eye camera is used in the first modification. 

The first modification is performed under the "panoramic 5Q 
image sensing mode" shown in FIG. 1. 

In FIG. 9, reference numeral 705 denotes an object; 700, 
a parameter extracting apparatus; 100, an object lens; and 
200, an illumination unit. 

Reference numeral 703 denotes a pad on which the object 55 
705 is placed, and respective characters, "A", "B", "C" and 
"D" are written in the four corners of the pad 703 as markers. 
The pad 703 serves as a background image as in the first 
embodiment, as well as provides information for posture 
recognition, which is specific to the first modification, to the 60 
parameter extracting apparatus 700. In other words, the 
parameter extracting apparatus 700 calculates its posture on 
the basis of the direction, distortion, and other information, 
of the image of the letters, "A", "B", "C" and "D", written 
on the pad 703. 65 

FIG. 10 is a block diagram illustrating a configuration of 
the parameter extracting apparatus according to the first 
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modification. It should be noted that, in FIG. 10, the units 
and elements identified by the same or similar reference 
numerals as the number parts of the reference numerals in 
FIG. 3 have the same functions and operate in the same 
manner. 

In FIG. 10, reference numeral 100 denotes the object lens 
which may consist of a zoom lens. Further, reference 
numeral 101 denotes an iris diaphragm; 102, an image 
sensor, such as CCD; 103, an A/D converter; 104, an image 
signal processor; and 105, an image separator for separating 
the object image from a background image. 

Further, reference numeral 740 denotes a posture detector 
which detects a posture of the parameter extracting appara- 
tus 700 on the basis of the direction, distortion, and other 
information, of the markers written on the pad 703. Refer- 
ence numeral 720 denotes an image processor for extracting 
three-dimensional shape information of the object 705 from 
the image signals and the posture information; 770, a 
focusing state detector which has the same function and 
operates in the same manner as the focusing state detector 
270 in the first embodiment, except the focusing state 
detector 770 has only one lens; 750, an image memory; 710, 
a system controller which controls the entire apparatus; and 
730, a memory provided in the system controller 710. 

Next, an operation of the first modification will be 
described with reference to a flowchart shown in FIG. 11. 

The processes at steps SI to S5 shown in FIG. 11 are 
performed on a single image signal as in the case of the 
processes shown in FIGS. 5A, 5B (first embodiment). 

The point which is different from the first embodiment is 
a method of adjusting the zoom ratio at step S22. More 
specifically, the pad 703 is placed at an appropriate position 
in advance to the image sensing operation, since the param- 
eter extracting apparatus 700 in the first modification detects 
its posture in accordance with the image of the pad 703. 
Thereafter, the image separator 105 performs correlation 
operation or template matching between a reference pattern 
of the markers (i.e., four letters, "A", "B", "C" and "D'\ on 
four corners as shown in FIG. 9) on the pad 703 and the 
current image of the pad 703. Then, an image of each marker 
is extracted, and a position detection signal of the image of 
the marker is outputted to the system controller 710. The 
system controller 710 controls the zoom controller 106 to set 
the focal length f so that the markers on the pad 703 are 
inside of a proper range of the field of view. At the same 
time, information about the focal length f is stored in the 
memory 730 in the system controller 710. 

Thereby, the entire view of the pad 703 can be always put 
within the field of view, thus the posture can be always 
detected on the basis of the shapes of the markers. More 
specifically, the markers are written on the four corners of 
the pad 703, and an operator puts the object 705 inside of the 
four corners in the first modification. Therefore, whenever 
the parameter extracting apparatus 700 can recognize all the 
four markers, the object 705 is within the field of view of the 
parameter extracting apparatus 700. 

The aforesaid processing is the contents of the "zoom 
ratio adjustment" at step S22. 

Note that, according to the principle of the first 
modification, four markers are not necessarily needed, and 
three markers may be enough as far as the object is placed 
inside of the three markers. Further, the markers are not 
limited to letters, and can be symbols. Furthermore, they do 
not have to be written on the pad 703, and can be in the form 
of stickers which can be replaced freely. 

After image sensing parameters for the optical system are 
set in the processes at the steps S2 to S22, an LED of the 
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display 240 is lit at step S23 to notify the user that the After the power is turned on at step SI, the user selects the 

apparatus is ready for input. type of the pad 703 by the pad selector 780 at step S30. Next, 

In response to this notification, the user starts inputting at tne system controller 800 reads out information on the 

step S24, and presses the shutter 230 at some intervals while selected pad from the memory 820 on the basis of the input 

moving the parameter extracting apparatus 700 at step S25, 5 selection information at step S31, then uses the information 

thereby inputting images. Upon this operation, the system for focal len S th contro1 ' focusin S contro1 ' P osture detection » 

controller 710 adjusts a focal length so that all the markers and so on. 

are always within an appropriate range of the field of view According to the second modification the user can select 

of the camera on the basis of information from the image a P? d to ^ ™* as a bac ^° und out of a plurality of pads 

separator 105 so that the object 705 is always in the field of 10 Wlth ***** ,hus i is possible to set the background which 

. r c . «-i i • ■ is most suitable to the shape and size, for example, of an 

view of the camera. Further, the image parameter object whose sfa ^ tQ £ e reco ized> ^ a ^ the 

information including the focal length, is stored in the image parameters which are most suitable to the 

memory 730 at each image sensing point. With this stored ob - )CcX can be determined. Therefore, the precision of the 

information, the posture detector 740 detects the posture of three-dimensional shape recognition on the basis of images 

the parameter extracting apparatus 700 from the detected ^ obtained by using the optimum image sensing parameters 

position of the markers (refer to FIG. 12). C an be improved. 

At the succeeding step S27, the image processor 720 Next, the three-dimensional recognition process accord- 
performs an image processing for the three-dimensional ing to the second modification will be described. The 
recognition. processes at steps S32 to S41 are the same as the processes 

More specifically, the image processor 720 reads out a 20 at step S2 to S27 explained in the first modification. More 

plurality of image signals (at each of image sensing points, specifically, the image sensing parameters are determined at 

Aq and Aj to AJ stored in the image memory 750. steps S32 to S36, then three-dimensional shape information 

Thereafter, it corrects read-out images and converts them of the object is extracted at steps S37 to S41. 

into images in the same focal state on the basis of the image In an image sensing processing performed at a plurality of 

sensing parameters stored in the memory 730 in the system 25 image sensing points at steps S3 9 to S45, there may be some 

controller 710, Further, the image processor 720 extracts cases in which the images sensed at specific image sensing 

three-dimensional shape information of the object 705 from points should be sensed again. In such a case, the user selects 

the corrected image signals and a posture signal obtained by a re -sensing mode through the pad selector 780. This selec- 

the posture detector 740, then outputs it to the recorder 250. tion is detected at step S42. The user must move the 

The recorder 250 converts the inputted signals into signals 30 parameter extracting apparatus 700 at a position where the 

of a proper format, then records them into a recording object is to be re-sensed. Then, the system controller 800 

medium. Images are inputted until the input at all the image makes the recorder 790 read images which have been sensed 

sensing point is completed at step S28, thereafter, the and recorded at step S43. Then, at step S44, the system 

process is ended at step S29. controller 800 performs matching operation between the 

35 currently sensed image and the plurality of read-out images, 

Second Modification and spe cifies an image to be replaced. 

FIG. 13 is a block diagram illustrating a configuration of When the area currently sensed and the read signals are 

a parameter extracting apparatus of a second modification. corresponded by the matching operation, then the LED, for 

In the second modification, a pad written with markers is 4Q example, in the display 240 is lit at step S37 to notify the 

used as in the first modification, however, it differs from the user that the apparatus is ready for input. Then at steps S38 

first modification in that the number of the pads used in the and S39, the previously sensed image is replaced by the 

second modification is plural. Further, the second modifi- currently input image. 

cation is further characterized in that image sensing opera- It should be noted that, when the object is re -sensed at step 

tion can be redone at arbitrary positions. 45 S42, it is possible to change the position of the object 705 

In FIG. 13, the same units and elements identified by the on the pad 703. In this case, too, the previously recorded 

same reference numerals as those in FIG. 2 (first image is matched to the currently input image, and input 

embodiment) and FIG. 10 (first modification) have the same process starts from a base point where the points in the two 

functions and operate in the same manner, and explanations images are corresponded. 

of those are omitted. 50 Further, in a case of terminating the input operation and 

In FIG. 13, reference numeral 820 denotes a memory for re-sensing images of the object, the recorder 790 reads out 

storing information on each of a plurality of pads. A user can me image sensing parameters in addition to a previously 

select the type of the pad 703 through a pad selector 780 recorded three-dimensional shape information and pixel 

(e.g., keyboard). Reference numeral 790 denotes a recorder signals, and input operation is performed after setting the 

which records three-dimensional shape information as well 55 sam e image sensing parameters to those previously used in 

as the image sensing parameters. This also has a function of me image sensing operation. 

reading out the stored information when necessary. Refer- Further, in a case of using a pad 703 which is not 

ence numeral 800 denotes a system controller having the registered in the memory 820, the pad 703 is to be registered 

same function as the system controller 210 in the first through the external I/F 280 from a computer, or the like, 

embodiment, and controls the entire apparatus. Reference 60 In a case where no image of the object is to be re -sensed, 

numeral 810 denotes a matching processor for performing a the processes at steps S39 to S42 are repeated until finishing 

matching process, based on pixel information, between inputting images, then terminated at step S46. 

three-dimensional shape information, recorded in advance, As explained above, the three-dimensional shape recog- 

which is read out by the recorder 790 and an image currently nition apparatus is featured by determination of the optimum 

sensed. 65 image sensing parameters and storage of images, and per- 

Next, an operation of the second modification will be forms three-dimensional shape extraction in high precision 

described with reference to a flowchart shown in FIG. 14. by using the optimum image sensing condition parameters. 



07/21/2004, EAST Version: 1.4.1 



US 6,6' 

17 

It should be noted that the units which are in the right side 
of a dashed line L can be configured separately from the 
parameter extracting apparatus of the first embodiment, and 
may be provided in a workstation, a computer, or the like. 

Next, a three-dimensional shape extraction apparatus, to 
which an image sensing apparatus of the present invention 
is applied, according to the second embodiment will be 
described. 

Second Embodiment 
Editing of a Three-dimensional Image 

A configuration of a three-dimensional image editing 
system to which the present invention is applied is described 
below. 

The three-dimensional shape extraction apparatus accord- 
ing to the second embodiment is for displaying and editing 
a three-dimensional image by applying a principle of the 
method of determining the image sensing parameters 
described in the first embodiment (i.e., the three- 
dimensional shape extraction mode, the display and editing 
mode and the panoramic image sensing mode which are 
shown in FIG. 1). 

Configuration of the System 

A configuration of the three-dimensional image editing 
system according to the second embodiment is explained 
below. 

FIG. 15 is a brief view of the three-dimensional shape 
extraction apparatus and the environment in which the 
apparatus is used according to the second embodiment. 

The system shown in FIG. 15 has an image sensing head 
(camera head) 1001, an image processing apparatus 4000, a 
monitor 1008, an operation unit 1011, a printer 1009, and 
programs 2000 and 3000 for combining data and editing a 
document. The image sensing head 1001 adopts multi-lens 
image sensing systems. Reference numerals 1100L and 
1100 R respectively denote left and right object lenses 
(simply referred as "right and left lenses", hereinafter), and 
1010L and 1010R denote image sensing areas for the right 
and left lenses llOOLand 1100R, respectively. Further, these 
image sensing areas have to be overlapped in order to obtain 
a three-dimensional image. 

Further, reference numeral 1002 denotes an object; 1003, 
a background stage to serve as a background image of the 
object 1002; and 1200, an illumination unit for illuminating 
the object 1002. The illumination unit 1200 illuminates the 
object in accordance with the image sensing environment. 

Reference numeral 1004 denotes a posture detector for 
detecting a posture of the image sensing head 1001 when 
sensing an image. The posture detector 1004 has a detection 
function of detecting a posture (includes posture 
information) of the image sensing head 1001 by performing 
image processes on the basis of information obtained from 
the background stage 1003 and another detection function 
for physically detecting a posture of the image sensing head 
1001 with a sensor, such as a gyro, or the like. 

The image sensing head 1001 senses the object 1002 
while moving from the starting point Aq for image sensing 
operation to the end point A„. Along with the image sensing 
operation performed at each image sensing point between 
the starting point Aq and the end point A„, the position and 
posture of the image sensing head 1001 are detected by the 
posture detector 1004, and detected posture information is 
outputted. 
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A memory 1005 stores image data, obtained by the 
camera head 1001, and the posture information of the 
camera head 1001, obtained by the posture detector 1004. 

A three-dimensional image processor (3D image 
5 processor) 1006 calculates three-dimensional shape infor- 
mation of the object on the basis of the image data stored in 
the memory 1005 and the corresponding posture information 
of the camera head 1001. 

A two-dimensional image processor (2D image 
10 processor) 1007 calculates two-dimensional image data of 
the object seen from an arbitrary viewpoint in a style 
designated by a user from the three-dimensional image data 
of the object obtained by the 3D image data processor 1006. 

FIG. 16 shows types of images processing prepared in the 
image processing apparatus according to the second embodi- 
ment. A user selects a type, in which an image is to be 
outputted, out of the prepared types of images shown in FIG. 
16 via the operation unit 1011 shown in FIG. 15. 
20 More concretely, a user can select whether to process an 
image of the object as a half-tone image (e.g., 1012 in FIG. 
16), or as a wire image in which edges of the object are 
expressed with lines (e.g., 1013 in FIG. 16), or as a polygon 
image in which the surface of the object is expressed with a 
25 plurality of successive planes of predetermined sizes (e.g., 
1014 in FIG. 16) via the operation unit 1011 which will be 
described later. 

The document editor 3000 is for editing a document, such 
as text data, and the data combining program 2000 combines 
30 and edits document data and object data obtained by the 2D 
image processor 1007. 

The monitor 1008 displays two-dimensional image data 
of the object, document data, and so on. 

The printer 1009 prints the two-dimensional image data of 
35 the object, the document data, and so on, on paper, or the 
like. 

The operation unit 1011 performs various kinds of opera- 
tions for changing viewpoints to see the object, changing 
styles of image of the object, and combining and editing 

40 various kinds of data performed with the data combining 
program 2000, for example. 

FIGS. 17A and 17B are block diagrams illustrating a 
detailed configuration of the image sensing head 1001 and 
the image processing unit (the unit surrounded by a dashed 

45 line in FIG. 15), which constitute a three-dimensional shape 
information extraction block together. 

In FIGS. 17A, 17B, the right and left lenses 1100R and 
1100L consist of zoom lenses. 

50 Functions of iris diaphragms 1101 R and 1101L, image 
sensors 1102R and 1102L, A/D converters U03R and 
1103L, image signal processors 1104R and 1104L, image 
separators 1105R and 1105L, zoom controllers 1106R and 
1106L, focus controllers 1107R and 1107L, iris diaphragm 

5S controllers 1108R and 1108L, the posture detector 1004, and 
so on, are the same as those in the first embodiment. 

A system controller 1210 corresponds to the system 
controller 210 in the first embodiment, and is for controlling 
the overall processes performed in the three-dimensional 

go shape extraction apparatus. The system controller 1210 is 
configured with a microcomputer 1900, a memory 1910 and 
an image processing section 1920, as shown in FIG. 18. 

An image processor 1220 corresponds to the image pro- 
cessor 220 explained in the first embodiment, and realizes 

65 functions of the memory 1005, the 3D image processor 
1006, and the 2D image processor 1007 which are shown in 
a schematic diagram in FIG. 15. More specifically, the image 
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processor 1220 extracts three-dimensional shape informa- image data of the object is obtained, the user is notified by 

tion of the object from image signals of the sensed object, an indicator which is for notifying the end of the image 

Further, it converts three-dimensional shape information of sensing (not shown) and the image sensing operation is 

the object into information in integrated coordinate systems completed. 

in accordance with posture information of the camera head 5 How the camera head 1001 moves and a method of 

at each image sensing point obtained by the posture detector inputting images are the same as those of the parameter 

1004. extracting apparatus described in the first embodiment. 
An detailed operation of a camera portion of the three- Next, the 3D image processor 1006 calculates to generate 

dimensional shape extraction apparatus according to the three-dimensional image data on the basis of the image data 

second embodiment will be described below with reference 10 and the posture information (posture and position of the 

to FIGS. 17A, 17B. camera head 1001 when sensing images) corresponding to 

Images of the object are inputted through the lenses each image data stored in the memory 1005. 
1100R and 1100L. The inputted images of the object are The 2D image processor 1007 calculates to obtain two- 
converted into electrical signals by the image sensors 1102R dimensional data of an image of the object seen from the 
and 1102L, The obtained electrical signals are further con- 15 image sensing point (Aq) from which the object is first 
verted from analog signals to digital signals by the AID sensed on the basis of the three-dimensional image data 
converters 1103R and 1103L, then enter to the image signal obtained by the 3D image processor 1006, and the monitor 
processors 1104R and 1104L. 1008 displays the calculated two-dimensional data. At this 

In the image signal processors 1104R and 1104L, the time, the 2D image processor 1007 converts the three- 
digitized image signals of the object are converted into dimensional data into the two-dimensional data in an image 
luminance signals and color signals of appropriate formats. type (refer to FIG. 16) selected via the operation unit 1101 
Then, the image separators 1105R and 1105L separate by the user. 

images of the object whose three-dimensional shape infor- Further, the three-dimensional shape extraction apparatus 

mation is subject to measurement from a background image 25 according to the second embodiment is able to change the 

on the basis of the signals obtained by the image signal image of the object displayed on the monitor 1008 to an 

processors 1104R and 1104L. image of an arbitrary image type designated via the opera - 

A method of separating the images adopted by the image tion unit 1011 by the user or to an image of the object seen 

separators 105R and 105L in the first embodiment is appli- &om an arbitrary viewpoint. More specifically, when an 

cable in the second embodiment. 30 ima g e tv P e is designated via the operation unit 1011, the 2D 

4 , . r »u u* • . *u image processor 1007 again calculates to obtain two- 

The separated images of the object enter the image j ? j • * ^ • . *u 

u j • • i u • r~ dimensional image data of a designated image type on the 

processor 1220 where three-dimensional shape information . _ t , & ,. . ,. & j4 ~ . , 

. . . j 4 u u • e *™ • f .—a basis of the three-dimensional image data. Further, in order 

is extracted on the basis of image sensing parameters used . . . . * , . , 

, . * . to change viewpoints, an image of the object 1015, as shown 

upon sensing the images of the object. . r . ,. f . 4 , m#u> 

r b & J 35 m FIG. 19 for example, displayed on the monitor 1008 can 

Process of Sensing Images be changed to any image of the object seen from an arbitrary 

viewpoint, as images denoted by 1016 to 1021 in FIG. 19. 

When a user operates a release button 1230 after facing The usef caQ designate l0 output the image data of the 

the camera head 1001 to the object 1002, operation to sense sensed object {Q , hc printer m9 after cnanging viewpoints 

images of the object is started. ^ or types of the image data according t0 purpose of 

Then, the first image data is stored in the memory 1005, using the image. Further, the user can also combine or edit 

In the three-dimensional image input mode, the user moves document data, made in advance, and image data, generated 

the camera head 1001 from the image sensing point Aq to the by the 2D image processor 1007, while displaying those data 

point A„ sequentially around the object. G n the monitor 1008. If the user wants to change image types 

After the camera head 1001 sensed at an image sensing 45 and/or viewpoints of the image of the object in this 

point A„ in a way from the point Aq to the point A„ and when combining/editing process, the user operates the operation 

the posture detector 1004 detects that the position and the unit 1011. 

direction of the camera head 1001 are changed by a prede- _ , . t . C1 e n 

. , , • * *u • * a Determination of Image Sensing Parameters 

termined amount comparing to the image sensing point A„, to & 

the next image sensing operation is performed at the next 50 A flowchart shown in FIG. 20 shows a processing 

image sensing point A m+1 . Similarly, images of the object sequence of the camera portion of the three-dimensional 

are sensed sequentially at different image sensing points shape extraction apparatus according to the second embodi- 

until the camera head 1001 reaches the point A„. While ment. 

sensing the images of the object as described above, the In FIG. 20, steps S101, S102, S104, S105, S106, S108 and 

amount of change in position and direction, obtained from S5 S109 are the same as the steps SI, S2, S4, S5, S6, S8 and S9 

image data as well as posture data from the detector 1004, described in the first embodiment, respectively, 
of the camera head at each image sensing point with respect Briefly, "exposure adjustment" at step S102 is for con- 

to the position Aq from which the camera head 1001 sensed trolling image signals so that luminance level of the image 

the object 1002 for the first time is stored in the memory signals is high enough for performing three-dimensional 

1005. 60 shape information extraction. 

It should be noted that, in a case where the posture "In-focus point adjustment" at step S105 is for controlling 

detector 1004 detects that at least either the position or the the aperture number (i.e., aperture diaphragm) of the camera 

direction of the camera head 1001 is greatly changed while so that an image of the object is within the depth of focus by 

the camera head 1001 moves from Aq toward A„, the adopting the same method explained with reference to FIG. 

apparatus warns the user. 65 6 in the first embodiment. 

The aforesaid operation is repeated a few times. Then, Further, "zoom ratio adjustment" at step S106 is for 

when enough image data for calculating three-dimensional adjusting the zoom ratio so that an image of the entire object 
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falls within the image sensing area of each image sensing the right optical system is stored in the memory 1073R, and 

system, as described with reference to FIGS. 7 A and 7B in the image currently being sensed by the left optical system 

the first embodiment. is stored in the memory 1073L. 

"Readjustment of Parameters and Adjustment of Right- The overlapping portion detector 1092 detects an over- 
Left Difference" at step S108 includes a correction process 5 lapping portion between the image, sensed by the right 

performed in a case where the focal length f is changed as optical system and stored in the memory 1073R, and the 

a result of the "zoom ratio adjustment" and the depth of image, sensed by the left optical system and stored in the 

focus is changed more than an allowed value as a result of memory 1073L. A template mapping which will be 

the zoom ratio adjustment, and a process to correct differ- explained later, for example, may be used for detecting an 
ences between the right and left lenses, as in the first 30 overlapping portion. 

embodiment, The electronic view finder in the second embodiment has 

"Setting of Resolving Power" at step S109 has the same a characteristic in the way of displaying images, 

purpose as step S9 in the first embodiment. As shown in FIG. 22, no image is displayed on the EVF 

After the image sensing condition parameters are adjusted 1240 for areas wnich are not overlapped between the right 

by the processes at steps S100 and S200 in FIG. 20, the and left images, and an overlapped portion of the images is 

system controller 1210 gives a signal to an electrical view displayed on the EVF based on the image sensed by the right 

finder (EVF) 1240 to notify the user of the end of setting the optical system. An example shown in FIG. 22 shows what is 

image sensing condition parameters. The EVF 1240 may be displayed on the EVF 1240 when an object, in this case, a 

a CRT, an LCD, or a simple display, such as an LED. cup, is sensed by the right and left lenses 1100R and 1100L. 

Further, sound may be used along with the display. In FIG - 22 > portions indicated by oblique hatching are 

non-overlapping portions between the right and left images, 

Finder When Inputting a Three-dimensional Image thus neither right nor left image is not displayed. Whereas, 

* , « i .i i ■ i j -i , . the central area, where a part of the cup is displayed, of the 

An operator checks the display, e.g., an LED, then starts , . , . , . . . 

u. i • ■ i u ■ *■ ♦* EVF shows the overlapping portion, and a right image is 

extracting three-dimensional shape information. 25 ^ i j . ■ ..... . . . fa . °. 

* , displayed to show that there is the overlapping portion 

When the operator presses an input start button (not between the right and left images, 

shown), a detection signal by the posture detector 201 is A ^ q&q obtaiQ a three . dimensional image without fail 

mi la ze . ky confirming that an object (either partial or whole) which 

As described above, the image processing apparatus in the tne user wants to three-dimensionally display is displayed on 
second embodiment can generate a three-dimensional image 30 me £yp 1240. 

and a panoramic image. A three-dimensional image can be It should be noted lhatj wheQ the USCf presses the rdease 

generated from more than one image sensed by a double- button 1230 aftef confirming the image displayed on the 

lens camera (e.g., by the image sensing head 1001 shown in EVF n40) two iXt| the right image and the left 

FIG. 15) or sensed by a single-lens camera at more than one are sensed at Qne point> aQd overlap . 

image sensing point. In either case, an overlapping portion ping portions of tne right and left images are compressed by 

between an image for the right eye and an image for the left using JpEG which is a compressing method, for example, 

eye of the user is necessary in order to observe the images and recorded< reason for storing on i y me overlapping 

of the object as a three-dimensional image. portions is to avoid storing useless information, i.e., the 

In order to sense an object by using a multi-lens camera non-overlapping portions, since a three-dimensional image 

(or sense from a plurality of image sensing points), G f the overlapping portion can be obtained, 

conventionally, it is necessary to provide two view finders. Note> in a case where the release buUon 12 30 is pressed 

Then, framing is performed by matching images of the when there is no overlapping portion in the angles of views 

object displayed on the two view finders. In the second of lne right ^6 left image sensing systems, the sound 

embodiment, it becomes possible to perform framing of the generator 1097 generates an alarm indicating that there is no 

images of the object with a single view finder. For this sake, correlation between the right image and the left image. The 

a finder part (EVF 1240) of the image sensing head 1001 is user may notice that there is no correlation between the right 

devised so that the overlapping portion can be easily image and the left iraagej however, when the user deter- 

detected. mines it is okay, the image sensing process can be continued 

Referring to FIGS. 17 and 21, an operation of the finder 5Q by further pressing the release button 1230. 

according to the second embodiment will be described. ^f ter a plurality of images of the object are sensed at a 

As shown in FIG. 21, the display operation on the finder plurality of image sensing points without any miss- 
in the second embodiment is performed with image memo- operation, the recorder 1250 is disconnected from the image 
ries 1073R, 1073L, 1075R and 1075L, the EVF 1240, an sensing apparatus and connected to a personal computer, 
overlapping portion detector 1092 and an sound generator 55 Thereby, it is possible to use the obtained information with 
1097 which are shown in FIGS, 17A, 17B. an application software on the personal computer. 

In a case of sensing the object from a plurality of image In order to use the information on a computer, an image 

sensing points and obtaining three-dimensional shape infor- is automatically selected by checking a grouping flags 

mation on the basis of the sensed images, the sensed images included in supplementary information of the images and 
are stored in a recorder 1250 as images relating to each other. 6 o displayed on the personal computer, thereby using the right 

The image memories 1073R, 1073L, 1075R and 1075L are and left images. 

used for primary storage of the sensed images. Especially, A recording format of an image in the recorder 1250 in the 

the last sensed images are stored in the image memories three-dimensional image input mode is shown in FIG. 23. 

1075R and 1075L, and the images currently being sensed are More specifically, posture information, image sensing con- 
stored in the memories 1073R and 1073L. 65 dition parameters, an overlapping portion in a right image 

In a case where the three-dimensional image input mode and an overlapping portion in a left image are stored for each 

is selected, the image of the object currently being sensed by image sensing point. 
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In the image files stored in the recorder 1250, group As other example of the second embodiment, an overlap- 
identifier/image sensing point/distinction of right and left ping portion between the right and left images may be 
cameras/compression method are also recorded as supple- obtained on the basis of parameters for the image sensing 
mentary information. apparatus, such as the focal lengths, distance to the object 
Panoramic Image Sensing 5 &om the ima g e sensing apparatus, the base line length, and 

A panoramic image sensing is a fiinction to synthesize a the convergence angle of the right and left image sensing 

plurality of images sensed at a plurality of image sensing systems instead of finding it by correlation between the right 

points as shown in FIG. 24, and more specifically, a function and kft una g es ' aD ™*& of the obtained overlapping 

to generate an image as if it is a single continuous image P ortlon 15 displayed. With this method, although precision 

with no overlapping portion and no discrete portion. 10 ma y dro P somewhat compared to obtaining the overlapping 

In the second embodiment, the image processing appara- P ortion with correlation between the right and left images, 

tus has the overlapping portion detector 1092 for detecting una S e memories can be saved, thus reducing manufacturing 

an overlapping portion. Thus, in the panoramic image sens- cosl * 

ing mode, the system controller 1210 stores images, sensed ^ . r « ~ . , T 

at a current image sensing point A„ and transmitted from the 35 Extraction of Three-Diroenstonal Information 

image signal processors 1104R and 1104L, in the memories Next, extraction of three-dimensional information accord- 

1073R and 1073L, and controls so that the image sensed at mg t0 tne second embodiment will be described. 

the last image sensing point are read out from tfie First, extraction of information on distances to a plurality 

recorder 1250 and stored in the memories 1075R and 1075L, e . , t , c A , ((J . ^ . 

p ~, . ♦ II iiin » i *u i of points on the object (referred by "distance image 

Further, The system controller 1210 controls the overlapping 20 • c »■ L • & \ *u u • r*u j- * 1 

portion detector 1092 to detect an overlapping portion ^rmation , hereinafter) on the basis of three -dimensional 

between the image sensed at a current image sensing point «W ° btained at a . sm S le ma e* s f ensm S P.^nt will be 

A^ and stored in the memory 1073R and the image sensed described. A processing sequence of extracting distance 

at the last image sensing point A m . 1 stored in the memory ima g e "^formation from three-dimensional images is shown 

1075R. The system controller 1210 further controls the 25 in FIG. 26. 

image processor 1220 so that the overlapping portion in the In FIG. 26, right images (R-images) 1110R and left 
image stored in the memory 1073R (i.e., the image currently images (L-images) 1110L are three-dimensional images 
being sensed) is not displayed. Then, the overlapping por- stored in the memory 1910 (shown in FIG. 18). Reference 
tion is not displayed on the EVF 1240 as shown in FIG. 25. numerals 1111R and 11111 denote edge extractors which 
While checking the image displayed on the EVF 1240, the detects edges from the stereo images 1110R and 1110L. 
user moves the image sensing head 1001 so that the over- A corre sponding edge extractor 1113 finds which edge 
lapping portion (i.e., the portion which is not displayed) corresponds to which edge in the stereo images 1110R and 
disappears, and thereafter presses the release button 1230. m0L , n other words u extracts correspondin g points 
In the panoramic image sensing mode in the second wn ich indicate a point on the object. A stereoscopic corre- 
embodiment, as described above, it is possible to obtain a 3$ sponding point extractor 1112 extracts corresponding points 
panoramic image with no fail. Note, in FIG. 25, a frame which indicate the same point on the object in the three- 
1500 shows a field of view seen at the last image sensing dimensional images 1110R and 1110L. 
point /y The user can obtain a panoramic image more The ^ correS pondence information on the point of the 
certainly by giving attention to the frame 1500. o5ject extracted by the two corresponding point extractors 
Further, in the panoramic image sensing mode, a series of 4Q m2 and 1113 have to be the same. An inconsistency 
obtained images are stored in the recorder 1250 with supple- eliminating unit 1114 determines whether or not there is any 
mentary information, as in the case of the three-dimensional inconsistency between the correspondence information 
image input mode. obtained by corresponding edge extractor 1113 and the 
According to the three-dimensional image input mode and correspondence information obtained by the stereoscopic 
the panoramic image sensing mode in the second embodi- 45 corresponding point extractor 1112. If there is, the obtained 
ment as described above, image sensing failure can be information on the corresponding points having inconsis- 
prevented by displaying an image on the EVF 1240 so that tency is removed. Note, the inconsistency eliminating unit 
an overlapping portion can be easily seen. 1114 ca n perform determination while weighing each output 
Furthermore, in a case where an image sensing operation from the two corresponding point extractors 1112 and 1113. 
is continued when there is no overlapping portion in the 50 An occlusion determination unit 1115 determines whether 
three-dimensional image input mode, or in a case where an there is any occlusion relationship found in two sets of 
image sensing operation is continued when there is an corresponding point information on the point of the object or 
overlapping portion in the panoramic image sensing mode, not by using position information on the corresponding point 
an alarm sound is generated, thereby further preventing information and an index (e.g., remaining difference 
failure of image sensing operation. 5S information) indicating degree of correlation used for find- 
Further, in a case where image sensing operations are ing the corresponding points. This increases the reliability of 
made to be related to each other as a group, the grouping the results of the corresponding point processing performed 
information is also recorded as supplementary information by the stereoscopic corresponding point extractor 1112 and 
of the recorded image, thus it is easier to operate on a the corresponding edge extractor 1113. Correlation coeffi- 
personal computer. 60 cients or remaining difference, as mentioned above, may be 
Further, in a three-dimensional image sensing mode, since used as the index indicating the degree of the correlation, 
only the overlapping portion between the right and left Very large remaining difference or small correlation coeffi- 
image is displayed on the EVF, a user can clearly see an cients mean low reliability of the corresponding relation - 
image which can be three -dimensionally observed. ship. The corresponding points whose correspondence rela- 
Furthermore, only the overlapping portion between the right 65 tionship has low reliability are dealt with as either there is an 
and left images is recorded, it is possible to prevent a waste occlusion relationship between the points or there is no 
of memory. correspondence relationship. 
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A distance-distribution processor 1116 calculates infor- In a case of using the Robert filter, the following equation 

mation on distances to a plurality of points on the object by may be used instead of the equation (11). 

using the triangulation from the correspondence relation- 4ij)^m-AW+i)}«*{flfriJhM+ty 
ship. The triangulation is as described in relation to the 

equation (7) in the first embodiment. s WheD usin S the Sobel filters > an x-direction filter f x and a y 

^ . . „ „ direction filter L are defined as below. 

Characteristic point detectors 1117R and 1117L confirm y 

identity of characteristic points (e.g., markers) on the back- 
ground 1003. A correction data calculation unit 1118 finds 
image sensing parameters (aperture diaphragm, focal length, 
etc.), a posture and displacement of the image sensing head 

by utilizing the characteristic points extracted from the /-i 2 -l 

background image by the characteristic point detectors / = o o 0 

1117R and 1117L. ' ( 1 2 1 

Next, an operation of the image processing apparatus 1S 

1220 will be described in sequence with reference to FIG. „„_.., c , ■ , « , 

2g When the slope of the edge is expressed by 9, then, 

First, a method of extracting corresponding points per- ,f , (]4J 

formed by the corresponding points extractors 1112 and 0 = tan ~ { yj x ) 

1113 will be described. In the second embodiment, a tern- 20 
plate matching method is used as the method of extracting 

corresponding points. The edge extractors further applies binarization process 
In the template matching method, a block (i.e., template) on the i^ges whose edges were enhanced to extracts edge 
of NxN pixel size, as shown in FIG. 27, is taken out of either P° rtl0n ^ ™ e binarization is performed by using an appro- 
ve right image 1110R or the left image 1110L (the right 25 V n *^ thresnold * . 

° 111An . * ■ > i l j. A \, ?. The image processing apparatus 1220 detects information 

image 1110R is used in the second embodiment), then the B / _ , 6 « T. . r • „ . ■ ^ \ 

. , i . u i- w f\a \a /kt xjt\ i on distances (referred as distance information , hereinafter) 

block is searched in a searching area 01 MxM (N<M) pixel v „ , ' . „ ' 

. ■ f . t . . , S ' ^ mm ;„ ,u fl^A m the distance-distnbuUon processor 1116 (shown in FIG. 

size in the other image (the left image 1110L in the second „ . . 4 ■ r *• • 

embodiment) for (M-N+l) 2 times. In other words, denoting 26 >- further combines the distance information in tune 

a point (a, b) as the point where the left-uppermost corner of 30 f ° UoWlng the ° rder ° f pK)CeSSed P ° mtS Sh ° Wn 

the template, T,, to be set, a remaining difference R(a, b) is . _ * * , . , ... . r tU 

calculated in accordance with the following equation, Nex !' » t^e-sequential combining .operation of the dis- 

tance information by the distance-distribution processor 
1116 is explained in more detail with reference to FIG. 28. 

h\ V V 1/ it; * t a m 35 The distance -distribution processor 1116 of the image 

processor 1220 operates distance information Z in accor- 
dance with the aforesaid equation (7). Since the image 
sensing operation is performed at each image sensing point 

The calculation is repeated, while the position (a, b) i n this case, the distance information Tf corresponding to 

is-moved inside of the image to be searched (the left image 40 image sensing points (Aq, A 3 , . . . , A„) forms a sequence of 

in the second embodiment), until a position (a, b) where the time distance information. Thus, the distance information is 

remaining difference R(a, b) is minimum is obtained. When denoted by Z'(i, j). If the image sensing operations at image 

the template image T^i, j) is at a position (a, b) where the sensing points are performed at an equal time interval St, for 

remaining difference R(a, b) is minimum, the central pixel the sake of convenience, the distance information can be 

position of the template T^i, j) is determined as a corre- 45 expressed with Z'(i, j), Z' +26f (i, j), Z' +3a '(i j), and so on, as 

sponding point. Note, in the above equation (10), \ Ri ^ t b ^(i j) shown in FIG. 28. 

is an image portion of the right image 1110R when the To the distance-distribution processor 1116, as shown in 

left-uppermost corner of the template is at a point (a, b). FIG. 26, the occlusion information is inputted from the 

The stereoscopic corresponding point extractor 1112 occlusion determination unit 1115, and image sensing 

applies the aforesaid template matching method on the 50 parameters and posture information are inputted from the 

stereo images 1110R and 1110L as shown in FIG. 26, thus correction data calculation unit 1118. 

obtaining corresponding points in luminance level. Referring to FIG. 28, a conversion unit 1121 maps the 

Extraction of corresponding points on edge is performed distance information Z'(ij) (1120) to integrated coordinate 

by applying the aforesaid template matching to the stereo systems. The distance information which is mapped to the 

images which are processed with edge extraction. The edge 55 integrated coordinate systems is combined. Here, the word 

extraction process (performed by the edge extractors 1111R "combine" includes a process of unifying the identical 

and 1111L) as a pre-processing for the extraction of corre- points on images of the object (unification process), an 

sponding points on edge enhances edge parts by using a interpolation process of interpolating between coordinates 

Robert filter or a Sobel filter, for example. of the obtained points, a determination process of determin- 

More concretely, in a case where the Robert filter is used, 60 m g reliability of coordinates of points on the basis of flags 

the stereo images 1110R and 1110L (referred by f(i,j)) are included in depth-of-focus information of the image sensing 

inputted the edge extractors 1111R and 1111L, then output- system, and selection or removal of distance information on 

ted as image data expressed by the following equation the basis of occlusion detection information, 

(referred by g(ij))» Combining the distance information always starts from 

65 the unification process. 

i&;>sqrt({/(v>/1^^ (ii) Referring to FIG. 28, the distance information 1120 of the 

obtained stereo images are generated in every second. 



07/21/2004, EAST Version: 1.4.1 



US 6,640,004 B2 

27 28 

Meanwhile, the system controller 1210 sends information, 1001, and the distance information projected on the projec- 

such as a displacement amount direction of the image tion planes is registered in the second embodiment. Further, 

sensing head 1001, to the processing unit 1116 in synchro- reference numerals 1810 to 1814 denotes central axes 

nization with the distance information. With the sent (optical axes) of the projection planes 1800 to 1804, respec- 

information, the obtained distance information is mapped to 5 tively. 

the integrated I coordinate systems by applying a processing ^ int ted coordinate systems are five coordinate 

method which will be explained later. The mapping to the . , . ' w . c 

* * _, , • ■ • systems (e.g., xyz coordinate systems) forming the aforesaid 

integrated coordinate systems is aimed at making it easier to £ v'rtual ro ection lanes 

combine the information which is obtained in every second. P ^ P * 

It is assumed that two corresponding points (xq, y 0 , Zq) io First > the distance information Z' l7 obtained as above is 

and (x lt y 1( z 2 ) are obtained from an image sensed at time projected on each of the projection planes (five planes). In 

t and an image sensed at time t+St. The determination the projection process, conversion, such as the rotation and 

whether these two corresponding points are an identical shifts, is performed on the distance information Z',y along 

point on an object or not is performed on the basis of the each of the reference coordinates. As an example, the 

following equation. When a small constant ^ is defined, if is projection process to the projection plane 1803 is shown in 

the following relationship, FIG. 30B. The same process as shown in FIG. 30B is 



fe-*i) 2 +(yo-j'i) 2 +(^i) 2 «i (15) 



performed for the projection planes other than the projection 
plane 1803. Further, the same projection process is per- 



is satisfied, then the two points are considered as an identical formed on the next distance information Z' + Then, the 

point, and the either one point is outputted on the monitor distance information is overwritten on each projection plane 

1008. m tmie sequence. 

Note, instead of the equation (15), the equation, As described above, distance information of an object 



^o-*i) 2 +Kyo-n) 2 +<^ 0 -*i) 2 «2 (i 6) 



along five base axes can be obtained. More concretely, one 
25 point may be expressed by five points, (x 0 , y 0 , z 0 ), (x 1? y l9 



can be used. In the equation (16), a, b, c and d are some z i)» ( x 2» Yi> z 2)* ( x 3» v 3» z 3) a °d (x 4 , y 4 , z A ). 

coefficients. By letting a«b«l and c«2, i.e., putting more A three-dimensional image is generated as described 

weight in the z direction than in the x and y directions, for above, 
example, the difference of the distances Z' in the z direction 

can be more sensitively detected comparing to the other 30 Correction of Image Sensing Parameters 
directions. 

Thereby, one of the corresponding points between the Image sensing parameters need to be corrected in 

images sensed at the image sensing points (Aq, A lf . . . f AJ response to a posture of the image sensing head 1001 with 

is determined. respect to the background stage 1003. Correction of the 

Upon combining the distance information, an interpola- 35 parameters is performed by the system controller 1210 with 

tion process is performed next, lne ne jp 0 f tne i mage processor 1220 or the posture detector 

The interpolation process in the second embodiment is a jqq^ 
griding process, i.e., an interpolation process with respect to 

a pair of corresponding points in images obtained at different In the second embodiment, there are two modes: one 

image sensing points (viewpoints). Examples of grids 40 mode is for correcting the image sensing parameters based 

(expressed by dashed lines) in the z direction are shown in on information from the posture detector 1004; and the other 

FIG. 29 as an example. mode is for performing the correction based on image 

In FIG. 29, 0 and • are a pair of extracted corresponding information from the image processor 1220. 
data, and □ is corresponding data obtained after interpolat- 

ing between • data and the O data on the grid by perform- 45 First > a metnod of correcting the image sensing param- 

ing a linear interpolation or a sprain interpolation, for eters on the basis of the ima S e information from the image 

example. processor 1220 will be explained with reference to FIG. 31. 

Upon combining the distance information, reliability I n the second embodiment, a pad is used as a background 

check is performed. st im that the pad 10 03 is irj m XYZ 

The reliability check is for checking reliability of coor- 50 coordinate ^ and a int on the pad 1003 is expressed 

dinates of corresponding points on the basis of information ^ ( y ^ {n ^ ^ s tem If lhe int 

on the depth of focus sent from the image sensing systems. * ^ ^ aD 

This operation is for removing corresponding point in for- \ a . . . u v ■ 5 u t a u~ ♦ .u 

t . * n i r u-r* u ■ *u - e *• *u an angle % about the Y axis, and by an angle 0<~ about the 

mation of low reliability by using the information on the „ ■ a JL i j u /it \) «/\ iu ♦* a- * 

, . cc • * * ii *■ Z axis, further slides by (U, V, W) with respect to coordinate 

depth of focus of the image sensing systems, as well as for 55 * c , f ; L V . 7 . r t t . 

selecting pixels on the basis of the occlusion information. s ^!f ms of » ch ° f the H '™S e %™ R $, s ? s ° ms ' lh ' n ™ 

TT 6h . . j. t . c . . , . arbitrary point on the pad 1003, (X, Y, Z), in the coordinate 

Upon combining the distance information, mapping to the 4 ; ' , , Ct . 1 • . Vv v t \ * 

. . . . i- . , r i • i « system of the left image sensing system, (X L , Y n Z f ), is, 

integrated coordinate systems is performed at last. J a & J v ^ 

A method of mapping distance information Z' to the 

integrated coordinate systems is shown in FIGS. 30A and 60 

30B. 

In FIGS. 30A and 30B, reference numeral 1002 denotes 
an object, and 1003 denotes a pad. The pad 1003 corre- 
sponds to a background stage to be serve as a background 

image. 65 Further, the arbitrary point on the pad 1003, (X, Y, Z), in 

Reference numerals 1800 to 1804 denote virtual projec- the coordinate system of the right image sensing system, 

tion planes of optical systems of the image sensing head (X^, Y^, Z^), is, 



'■A L B L C L \ Y 



(17) 
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thus, 



(25J 



Note, the A^, B if C L in the equation (17) are matrices which 
represent affine transformation, and they are defined by the 
following matrices. 



0 ' 
0 cosO A siu0 A 
0 -sinfl* cosBa , 

COS0S 0 -SU30B 1 

0 10 
s'usBs 0 COS#g J 

( cosOc sinflc 0 
sinfc cosflc 0 
0 0 1 
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,Zr\ 
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B' 


(A l -B l -C l -Ar'Br'Cr)' 


Yl 


-Ur 




0 




.z Li 


{Zr J 




,0, 



is obtained. 

Now, assume that coordinates of corresponding points of 
an arbitrary point P 0 (x, y, z) of the object in the right and left 
image sensing systems have been extracted by the charac- 
teristic point extraction process and the corresponding point 
in an image sensed with the left image sensing system is 
expressed by p^x^, y^ and the corresponding point in an 
image sensed with the right image sensing system is 
expressed by p£x r , y r ). Then, the position of the correspond- 
ing points, (u, v), in a coordinate system of the CCD of the 
image sensing head 1001 is, 



(«. v )=(*/» yi)-(x„ y,) 



where, 



. , Mi. *L 



Xl Yl\ 
Yl' Z l I 

Xr Yr\ 



v , Mff TR 



For example, if there is no convergence angle between 
multiple image sensing systems (1100R and 1100L), then, 



10 



(19) 



The matrices, A R , B Ri C R , in the equation (18) are also 
defined by the same matrices (19). 

For example, in a case where the image sensing head 1001 
is at a distance B from the pad 1003 along the X axis (with 
no rotation), A=B~C=1. Therefore, the following equation 
can be obtained from equations (17) and (18), 



(20) 



30 



35 



Accordingly, in a case where the pad 1003 rotates with 
respect to the image sensing systems, 



(21) 40 



45 



50 



(22) 

(23) 
(24) 



60 
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A, 



(Identity matrix) 



^R-BR-CR-E (Identity m>nix) 

U-B, V-W-0, 



and the equation (20) holds, therefore, 

(«,v) = (£)•(* 0) 



(26) 



20 Thus, the coordinates of the arbitrary point on the pad 1003 
in the Z direction becomes, 



(27) 



25 



With the equations as described above, the relationship 
between the pad and the image sensing systems are defined. 

The aforesaid is just a brief explanation of correcting 
image sensing parameters, however, the generality of the 
method is fully explained. Further, the details of this method 
is explained in the Japanese Patent Application Laid-Open 
No. 6-195446 by the same applicant of the present inven- 
tion. 

Further, in a case where correcting the image sensing 
parameters on the basis of outputs from sensors is applied, 
it is preferred to use the average of the outputs from the 
image processing units and the outputs from the sensors. 

As another choice of the method, it may be possible to 
shorten processing time required for image processing by 
using the outputs from the sensors as initial values. 

Editing a Three-dimensional Image 

Next, a combining process of stereoscopic information, 
image texture and a document file, and an output process 
will be explained. 

FIG. 33 shows a processing sequence of combining the 
stereoscopic information which is obtained as described 
above, the image texture, and the document file. Each 
process shown in FIG. 33 is performed by the image 
processor 1220. 

The extracted distance information (Z 1 ) is obtained 
through image sensing operations with a camera, thus, it 
often represents a shape different from a real object. Or, 
there are cases in which the distance information is not 
preferred because it represents the same shape as the real 
object. Referring to FIG. 33, a fitting processor 1160 corrects 
the extracted distance information {71) 1600 by using a 
distance template 1162 in response to a user operation. 

A paste unit 1161 pastes image texture 1602 to distance 
information which is corrected by the fitting process. 

A file combining processor 1163 integrates the corrected 
distance information 1601, the image texture 1602 and a 
document file 1603 to generate a single file. The document 
file 1603 is a document text inputted from the operation unit 
1101. The combined file is outputted as a two- or three- 
dimensional image. 
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First, a fitting process is explained. Further, reference numeral 2102 denotes a calibration pad 

A flow of the fitting process is shown in FIG. 34. In FIG. a °d the three-dimensional shape extraction apparatus 2100 

34, reference numeral 2501 denotes an image represented by detects the posture of itself on the basis of an image of the 

the extracted distance information; 2500, a model image to P a d 2102. 

be a template; and 2502 and 2503, differences between the 5 Note, characters, A, B, C and D written on the pad 2102 

extracted image 2501 and the model image 2500. are markers used for detection of the posture. The posture is 

The fitting processor 1160 first displays the aforesaid two calculated from the direction and distortion of these markers, 

images (2500 and 2501) on the monitor 1008 as shown in FIG. 37 is a block diagram illustrating a configuration of 

FIG. 34 and prompts a user to perform a fitting operation. If the three-dimensional shape extraction apparatus 2100 

a user designates to perform the fitting process with the 10 according to the first modification of the second embodi- 

model image 2500, then the fitting processor 1160 calculates ment. In FIG. 37, the units and elements which have the 

the difference 2502 between the two images, then corrects an same reference numerals as those in FIGS. 17A, 17B have 

image based on the differences 2502 into an image of the the same function and operation, thus explanation of them is 

uniform differences 2503. Further, the image 2501 is cor- omitted. 

reeled on the basis of the image of the difference 2503 into 15 A posture detector 3004 is for detecting the posture of the 

an image 2504. Note, the correction can be performed by three-dimensional shape extraction apparatus 2100 on the 

using an input pen. basis of the direction, distortion, and so on; of the markers 

As described above, a process of pasting the texture image written on the pad 2102. Reference numeral 3220 denotes an 

1602 to the corrected distance information 1601 obtained by 2Q image processor which extracts three-dimensional shape 

performing the fitting process is the same as a method which information of the object from image signals and posture 

is used in a field of computer graphics, and the like. information from the posture detector 3004. Reference 

Thereafter, the pasted image is further combined with the numeral 3210 denotes a system controller which controls the 

document file to generate a file for a presentation, for overa11 °P" a ^ n of lhe ^e-dimensional shape extraction 

example. 25 apparatus 2100. 

FIG. 35 shows a method of combining an image and a Operation of the three-dimensional shape extraction appa- 
document by the file combining processor 1163. In this ratus according to the first modification of the second 
method, information on an area in a document 1901 where embodiment will be explained next. FIG. 38 is a flowchart 
an image is to be embedded is stored in an image coordinate s ^ 0Win S a Processing sequence by the three-dinaensional 
information field. In an example shown in FIG. 35, the 30 shape extraction apparatus 2100 according to the first modi- 
image file 1902 is pasted at a coordinate position stored in ^ hon c of th u e secon ^ embodiment. The first modification 
a field 19008, and an image file 1903 is pasted at a duTers from the second emboo^ment m a method of adjustmg 
coordinate position stored in a field 19009. the zoom ratio. The apparatus of the first modification 

„ „ ^ , t nnM . , , _ . performs posture detection in accordance with characteristic 

Fields 19001 and 19002 in the document file 1901 contain ^ts ( . markers) of ^ d 2102> thus an . of d 

link information and respectively mdicate a connection 35 2i 02 is necessarily sensed in an appropriate area of the field 

relationship between the document file 190 and each of the f { f h { { { { ; 

image files 1902 and 1903. Further, fields 19003, 19005 and ODerat ion 

19007 are for document data. Fields 19004 and 19006 are p _ " 4 - . 
r . „•« • * Therefore, an image separator 3105 performs correlation 
for image insertion nags which indicate where images are to . ' 4 u 
be inputted, and refer to link information, in the fields 19001 40 operation or a template matching process be ween charac- 
and 19002, showing links to the image files. «" S,1C f °f!,tion (inputted in advance) of the markers 
' „f . . ,. . „ . . (letters, A, B, C and D) on the pad 2102 and image signals 
The document file is made with link information, image which are currenlly being in p U t te d, and detects the positions 
insertion flag and document data. The image file is embed- ot the markers . Thereafter, a result of the detection is 
ded in the position of the image embedding flag. Since the 0Ut p Utted t0 the system controller 3210. The system con- 
image file includes image coordinate information, an image UoUet 3210 sets the focal length of the image sensing system 
is converted into an image seen from an arbitrary viewpoint on the basis of the detec(ed positions of the markers so thal 
on the basis of the image coordinate information and embed- , he d 2102 ^ xnsed ^ an a pp ropria i e range of the fie i d of 
ded in practice. In other words, the image coordinate infor- view of lne . ^ tem A , , he same timej 
■nation is information i showing conversion relationship with information on lhe focal length which enables the field of 
respect to an originally obtained image. The image file thus view of the ^ ^ system t0 include the entire pad 
generated is finally used. 2102 is stored in a memory (no , shown) ^ the system 

First Modification of the Second Embodiment antioter 3210 ' V 1 ^!' ? " * 

entire pad in the field or view of the image sensing system, 

Next, a modification of the second embodiment will be 5S as well as to detect the posture of the three-dimensional 

explained. shape extraction apparatus 2100 on the basis of distortion of 

FIG. 36 is a brief overall view of an image processing the image of the markers, 

system according to a first modification of the second As shown in FIG. 38, when parameters for the image 

embodiment. In the first modification, the three-dimensional sensing system are set, then an LED of the EVF 1240 is 

shape extraction apparatus of the three-dimensional image 60 turned on to notify a user that the apparatus 2100 is ready for 

editing system of the second embodiment is changed, and input. 

this modification corresponds to the first modification of the i n response to this notification, the user starts inputting, 

first embodiment. and presses the release button 1230 at a predetermined 

In FIG. 36, an object 2101 is illuminated by the illumi- interval while moving the apparatus 2100, thus inputs 

nation unit 1200, and the three-dimensional shape informa- 65 images. At this time, the system controller 3210 sets the 

tion is extracted by a three-dimensional shape extraction focal length so that the markers on the pad 2102 with the 

apparatus 2100. object are always within an appropriate range of the field of 
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view of the image sensing system on the basis of information 
from the image separator 3105. Furthermore, information on 
the image sensing parameters, including the focal length at 
each image sensing point, is stored in the memory 1910. 
Accordingly, the posture detector 3004 detects the posture of 
the apparatus 2100 from the states of the markers. 

The image processor 3220 reads out a plurality of image 
signals stored in the image memories 1073 and 1075, then 
converts the image into an image of a single focal length on 
the basis of the information on the image sensing parameters 
stored in the memory in the system controller 3210. Further, 
the image processor 3220 extracts three-dimensional shape 
information of the object from the corrected image signals 
and the posture signal obtained by the posture detector 3004, 
then outputs the information to the recorder 1250. The 
recorder 1250 converts the inputted signals into signals of a 
proper format, then records them on a recording medium. 

Second Modification of the Second Embodiment 

FIG. 39 is a block diagram illustrating a configuration of 
a three-dimensional shape extraction apparatus according to 
the second modification of the second embodiment. The 
second modification of the second embodiment corresponds 
to the second modification of the first embodiment. This 
second modification is characterized in that images can be 
re-sensed by using a plurality of pads similar to the ones 
used in the first modification. 

Referring to FIG. 39, information on the plurality of pads 
is stored in a memory 4400. The I/F 1760 to an external 
device connects to a computer, or the like, for receiving 
information. The kinds of pads can be selected through the 
I/F 1760. 

The recorder 1250 stores three-dimensional shape infor- 
mation along with the image sensing parameters. It also has 
a function of reading out stored information when necessary. 
Reference numeral 4210 denotes a system controller which 
controls the overall operation of the entire apparatus of the 
second modification. 

A matching processor 4401 specifies an image to be 
re-sensed out of the images stored in the recorder 1250. 
Therefore, the matching processor 4401 searches the same 
image as the one which is currently sensed, from the images 
stored in the recorder 1250 by using a matching method. 

Next, an operation of the three-dimensional shape extrac- 
tion apparatus of the second modification of the second 
embodiment will be explained. The flow of the operation 
according to the second modification is shown in FIG. 38. 

In the apparatus according to the second modification, a 
user selects a kind of pad to be used when starting inputting. 
The system controller 4210 reads out information indicating 
characteristics of the selected pad from the memory for pads 
4400 in accordance with the information on designation to 
select a pad. 

Then, as shown in the flowchart in FIG. 38, the similar 
processes as in the first modification are performed to start 
inputting images of an object, then its three-dimensional 
shape information is extracted. Here, if the user wants to 
re-sense an image, then selects a re-sensing mode through 
the I/F 1760, 

Then, the controller 4210 sequentially reads out images 
which have been recorded by the recorder 1250, and controls 
the matching processor 4410 to perform a matching process 
between the read-out images and an image which is cur- 
rently being sensed. 

When the correspondence is found between the image 
which is currently being sensed and the read -out image in 
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the matching process, an LED of the EVF 1240 is turned on 
to notify the user that the apparatus is ready for input. 

Note, upon re -sensing an image, it is possible to change 
the position of the object 2101 on the pad 2102. In such a 
5 case, the matching process is also performed between the 
image which has been recorded and an image which is 
currently being sensed, then the matching image which was 
sensed before is replaced with the image which is currently 
being sensed. 

20 Further, in a case of terminating the input operation and 
starting over from the beginning of the input operation, the 
recorder 1250 reads out the three-dimensional shape infor- 
mation and image signals as well as image sensing param- 
eters which have been recorded, then the input operation is 

15 started by setting the image sensing parameters to the same 
ones used in the previous image sensing operation. 

Further, if the user wants to use a pad 2120 which is not 
registered in the memory 4400 in advance, then information 
of the pad 2120 is set from a computer, or the like, through 
the I/F 1760. 

Third Modification of the Second Embodiment 

It is possible to operate in the three-dimensional image 

25 sensing mode in addition to the three-dimensional shape 
information extraction mode by using the image sensing 
systems explained in the second embodiment. In other 
words, it is possible to provide images to be seen as 
three-dimensional images by using a plurality of image 

30 sensing systems. 

It is possible to select either the three-dimensional image 
sensing mode or the three-dimensional shape information 
extraction mode by using the external input I/F 1760. 
An operation of the image sensing apparatus in the 

35 three-dimensional image sensing mode is described next. 
In a case where the three-dimensional image sensing 
mode is selected through the I/F 1760, images sensed by the 
right and left image sensing systems are outputted. 

40 Further, since an overlapping portion in the right and left 
images can be obtained by calculating correlation between 
the images stored in the memories 1073R and 1073L, 
non-overlapping portions are shown in a low luminance 
level in the EVF 1240 as shown in FIG. 41 so that the 

45 overlapping portion corresponding to the image sensed by 
the right image sensing system can be distinguished. In FIG. 
41, when an object on a table is to be sensed, both right and 
left end portions of the image on the EVF 1240 are 
expressed in a low luminance level. Since a ball is in the left 

50 end portion of the image which is expressed in the low 
luminance level, a user can easily recognize that it can not 
be seen as a three-dimensional image. 

In contrast, the user can easily know that a tree and a plate 
in the central portion of the image can be displayed as an 

55 three-dimensional image. Thus, the portion which can be 
displayed as a three-dimensional image is seen clearly. 
Then, as the user presses the release button 1230, the right 
and left images are compressed in accordance with JPEG, 
and recorded by the recorder 1250. 

60 Fourth Modification of the Second Embodiment 

It may be considered to make the EVF 1240 as an optical 
system in order to provide the image sensing apparatus at 
low price. With an optical finder, it is impossible to display 
65 an image which has been sensed previously on it. 

Furthermore, since an image sensing area and an obser- 
vation area of an optical finder do not match if the optical 
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finder is not a TTL finder, there is a possibility to fail in an adjustment means for adjusting the image sensing condi- 

image sensing operation, because the user may not notice an tion parameter for at least one of said optical systems 

overlapping area even though there is the one. of said two image sensing means on the basis of images 

The fourth modification is for providing a variety of of the common subject sensed by said two image 

functions described in this specification at low cost. More 5 sensing means respectively, and in response to an 

concretely, an LED is provided within or in the vicinity of 0Ut P ut ° f said dominating means, so that the com- 

the field of view of an optical finder, and the LED is tuned T^m f K locat ? d > h lls d f epth ' wU , hin a deplh 

j rp * . c i ** °f field of the optical system of respective image 

on and off in accordance with output from a correlation r J r & 

, „ . . tt- i sensing means, 

detector. For example, in a case where there is an overlap- i to, ♦ j- * 1*1 u • u 

. T . , . L -.n 2. The apparatus according to claim 1, wherein each 

ping area, the LED is turned on, whereas there is not, the 10 " . ° 

r ™ j r™ , , . optical system of said two image sensing means has a zoom 

LED is turned on. Thereby, the image sensing apparatus can , , ,. 4 t • i i c n , u (( - 

be rovided at low ice adjustment means includes focal length setting 

e provi e a ow price. means for setting focal lengths of the zoom lenses of said 

Further, a plurality of LEDs may be provided both in the ^ image me ans for which image sensing condition 

X and Y directions, and the LEDs in the overlapping portion parameter is subject to adjustment by said adjustment 

are turned on. In this manner, not only the existence of any meanSj t0 a first position which & between a second position 

overlapping portion but also ratio of the overlapping portion on a sur f ace 0 f tne common subject which is closest to the 

to the display area can be recognized, making it easier to image scnsing means and a third position on a surface of the 

aotlce - common subject which is farthest from the image sensing 

Furthermore, by making a frame of the field of view of the means, whereby images of the common subject fall within 

optical finder with a liquid crystal, an overlapping portion the focal depths of the zoom lenses, 

can be identified more precisely than using the LEDs. 3. The apparatus according to claim 2, wherein each 

Fifth Modification of the Second Embodiment °P tical s y stem of said ^ ima 6 e "n*"* means further 

comprises an ins diaphragm, and said adjustment means 

A scheme of extracting three-dimensional shape informa- further comprises- 

tion and a basic structure of the apparatus are the same as ^ ^ fof calculating focal depths of lhe zoom 

those shown in FIG. 15. lenses of said ^ image j^^g means for wnich 

However, the image sensing systems do not have a sensing condition parameter is subject to adjust- 

function to input three-dimensional shape information, and menl by said adjustment means when the focal lengths 

the inputting operation may be performed by executing an of the zoom lenses are set to the first position, and 

image input program installed in a computer, for example. ^ di h control means for the iris 

Upon executing the input program, a user places an object diaphragms so that a range of the focal depths is 

to be measured on a pad. substantially between the second position and the third 

Then, in response to a command inputted by the user to position, 

execute the image input program installed in the computer 35 4 a pp ara tus according to claim 2, wherein the first 

from an input device, a window (referred as "finder position is the middle point between the second position and 

window", hereinafter) which corresponds to a finder of a tne tQ j rd position. 

camera is generated on a display device of the computer. 5 ^ apparams acc0 rding to claim 1, wherein said 

Then, when the user turns on the switch of the camera, an adjustment means includes means for selecting an image 

image sensed by the camera is displayed in the finder 4Q sensing condition parameter determined for either one of the 

window. optical systems of said two image sensing means, when the 

The user performs framing while watching the displayed image sensing condition parameters determined for said two 

image so that the image of the object is displayed in about i mage sensing means are different from each other, 

the center, then presses the shutter. Thereafter, the image of 6. The apparatus according to claim 1, wherein image 

the object is scanned to obtain image data of the object. The 45 sensing condition parameters include a diameter of an 

image data is processed by a processing apparatus which is aperture diaphragm and a focal length of the optical system 

exclusively for a camera, and by other processing apparatus, 0 f sa id two image sensing means. 

thereby three-dimensional data of the object is obtained. 7, The apparatus according to claim 1, wherein said 

The present invention can be applied to a system consti- adjustment means includes means for changing the focal 

tuted by a plurality of devices, or to an apparatus comprising 50 depths of the optical systems of said two image sensing 

a single device. Furthermore, the invention is applicable also means so that the common subject fails within the focal 

to a case where the object of the invention is attained by depths of the optical systems of said two image sensing 

supplying a program to a system or apparatus. means. 

The present invention is not limited to the above embodi- 8. The apparatus according to claim 1, wherein said 

ments and various changes and modifications can be made 55 adjustment means includes means for controlling angles of 

within the spirit and scope of the present invention. There- view of the optical systems of said two image sensing means 

fore to appraise the public of the scope of the present so that the common subject is in both the image sensing 

invention, the following claims are made. areas of said two image sensing means. 

What is claimed is: 9. The apparatus according to claim 1, wherein said 

1. An image sensing apparatus for sensing a common 60 adjustment means includes: 

subject whose three-dimensional image is to be generated, depth-of-focus changing means for changing focal depths 

comprising: of the optical systems of said two image sensing means 

at least two image sensing means, each having an optical so that the common subject falls within the focal depths 

system for sensing the common subject; of the optical systems of said two image sensing means; 

discriminating means for discriminating a difference 65 angle-of-view control means for controlling angles of 

between image sensing parameters of each optical view of the optical systems of said two image sensing 

system of said at least two image sensing means; and means so that the common subject is in both the image 
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sensing areas of said two image sensing means after 
said depth-of-focus changing means has changed the 
depths of focus; and 
means for reinitiating said depth-of-focus changing 
means and said angle-of-view control means in a case 5 
where a focal length set by said depth-of-focus chang- 
ing means is changed by said angle-of-view control 
means. 

10. The apparatus according to claim 1, further compris- 
ing position detection means for detecting the position of ]0 
each of said two image sensing means in synchronization 
with a user pressing a shutter, 

wherein said two image sensing means move together. 

11. The apparatus according to claim 10, further compris- 
ing means for notifying the user of a timing to press the 15 
shutter on the basis of detection by said position detection 
means. 

12. The apparatus according to claim 10, wherein the 
apparatus notifies the user whether or not a moving speed of 
said image sensing means, which is determined on the basis 20 
of detection by said position detection means. 

13. The apparatus according to claim 1, further compris- 
ing storage means for storing the image sensing condition 
parameters determined by said adjustment means, 

14. The apparatus according to claim 13, further com- 25 
prising detection means for detecting a posture of each of 
said image sensing means, 

wherein posture information detected by said detection 
means is stored as an image sensing parameter in said 3Q 
storage means. 

15. An image sensing apparatus for sensing a common 
subject whose three-dimensional image is to be generated 
from a plurality of images taken at a plurality of image 
sensing points, comprising: 

a plurality of image sensing means having an optical 
system for sensing the common subject; 

moving means for sequentially moving said image sens- 
ing means to at least two image sensing points; 

discriminating means for discriminating a difference 
between image sensing parameters of each optical 
system of said two image sensing means; and 

adjustment means for adjusting the image sensing condi- 
tion parameter for said image sensing means on the 
basis of a plurality of images of the common subject 
sensed by said image sensing means at said two image 
sensing points, in response to an output of said dis- 
criminating means, so that the common subject is 
located, through its depth, within a depth of field of said 
image sensing means set at a respective image sensing 
point. 

16. The apparatus according to claim 15, wherein said 
adjustment means includes: 

search means for searching a plurality of image portions, 
each of which has a predetermined pattern, within the 
plurality of images sensed at said two image sensing 
points; and 

determination means for determining whether or not the 
plurality of searched image portions are within the 
image sensing area of said image sensing means, 

whereby said adjustment means checks that images of the 
common subject fall within the image sensing areas of 
said image sensing means at said two image sensing 
points. 

17. The apparatus according to claim 16, further com- 
prising means for subjecting an image to focal length 
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correction process by changing the image sensing param- 
eters on the basis of the plurality of sensed images, 

18. The apparatus according to claim 16, wherein said 
determination means controls a focal length so that the 
searched patterns fall within a field of view of said image 
sensing means. 

19. The apparatus according to claim 15, wherein said 
adjustment means comprises: 

means for storing plural kinds of predetermined patterns; 
and 

means for selecting a pattern out of the plural kinds of 
patterns. 

20. The apparatus according to claim 15, further com- 
prising means for initiating to sense an image of the common 
subject by said image sensing means. 

21. An image processing apparatus which senses a com- 
mon subject with a plurality of image sensing means at a 
plurality of image sensing points and outputs three- 
dimensional shape information and image data information 
on the common subject, said apparatus comprising: 

conversion means for converting the image data informa- 
tion outputted by the image sensing means into an 
image which is observed the common subject from a 
desired viewpoint and which is of a predetermined 
type, and storing the image as a predetermined file 
format, the conversion being made on the basis of the 
three-dimensional shape information; and 
discriminating means, wherein each of said plurality of 
image sensing means comprises: 
an optical system for sensing an image; 
adjustment means for adjusting the image sensing 
condition parameters for the optical system of each 
image sensing means, in response to an output of 
said discriminating means, so that the common sub- 
ject is located, through its depth, within a depth of 
field of the optical system at an image sensing point; 
and 

a monitor for displaying an image sensed in the image 
sensing area of each of said plurality of image 
sensing means, 

wherein said discriminating means discriminating a 
difference between image sensing parameters of said 
optical system corresponding to the images of the 
plurality of image sensing points respectively. 

22. The apparatus according to claim 21, wherein said 
conversion means converts three-dimensional image data, 
obtained by sensing the subject at the plurality of image 
sensing points, into two-dimensional image data. 

23. An image processing apparatus which senses a subject 
with image sensing means at a plurality of image sensing 
points and outputs three-dimensional shape information and 
image data information on the subject, said apparatus com- 
prising: 

position detection means for detecting positions of said 
image sensing means over the plurality of image sens- 
ing points; 

operation means for operating the three-dimensional 
shape information, on the basis of image data repre- 
senting images obtained with said image sensing means 
and position data representing positions of said image 
sensing means at which the images have been sensed 
by said position detection means; 

conversion means for converting the images of the subject 
sensed at the plurality of image sensing points into 
image data information representing an image of the 
subject seen from an arbitrary viewpoint, on the basis 
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of the three-dimensional shape information on the 
subject operated by said operation means and forming 
an image data file based on a computer file format 
comprised of the converted image data information; 
and 

linking means for linking the image data file converted by 
said conversion means with data of another data file. 

24. The apparatus according to claim 23, wherein said 
image sensing means comprises: 

an optical system for image sensing; 

adjustment means for adjusting image sensing condition 
parameters for said optical system so that an image of 
the subject falls within an image sensing area and 
within the focal depths of said optical system at the 
plurality of image sensing points; and 

a monitor for displaying an image sensing area said image 
sensing means. 

25. The apparatus according to claim 23, wherein said 
conversion means converts three-dimensional image data, 2 o 
obtained by sensing the subject from the plurality of image 
sensing points, into two-dimensional image data. 

26. The apparatus according to claim 23, wherein the 
other file is a document file. 

27. The apparatus according to claim 23, wherein said 2 s 
position detection means stores information on background 

of the subject. 

28. An image processing apparatus which senses a subject 
with image sensing means from a plurality of image sensing 
points and outputs three-dimensional shape information and 30 
image data information on the subject, said apparatus com- 
prising: 

position detection means for detecting positions of said 
image sensing means over the plurality of image sens- 
ing points; 35 

operation means for operating the three-dimensional 
shape information on the basis of a plurality of images 
obtained at the plurality of image sensing points and the 
positions, corresponding to each of the sensed images, 
which are obtained by said position detection means, of 40 
said image sensing means; 

conversion means for converting the images of the subject 
sensed from the plurality of image sensing points into 
an image of the subject seen from an arbitrary view- 
point on the basis of the three-dimensional shape 45 
information on the subject operated by said operation 
means and forming an image data file comprised of the 
converted image; 

linking means for linking the image data file of the 
converted image and another data file based on a 
computer file format; and 
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display means for displaying an image of the image data 
file outputted from said combining means in a stereo- 
scopic manner. 

29. An image processing apparatus which senses a subject 
with image sensing means from a plurality of image sensing 
points and outputs three-dimensional shape information and 
image data information on the subject, said apparatus com- 
prising: 

position detection means for detecting change in the 
position of said image sensing means over the plurality 
of image sensing points; 

operation means for operating the three-dimensional 
shape information on the basis of a plurality of images 
obtained from the plurality of image sensing points and 
the positions, corresponding to each of the sensed 
images, which are obtained by said position detection 
means, of said image sensing means; 

conversion means for converting the images of the subject 
sensed from the plurality of image sensing points into 
an image of the subject seen from an arbitrary view- 
point on the basis of the three-dimensional shape 
information on the subject operated by said operation 
means and forming an image data file based on a 
computer file format comprised of the converted image; 
and 

linking means for linking the image data file of the 
converted image and another data file. 

30. An image processing apparatus which senses a subject 
with image sensing means from a plurality of image sensing 
points and outputs three-dimensional shape information and 
image data information on the subject, said apparatus com- 
prising: 

position detection means for detecting change in the 
position of said image sensing means over the plurality 
of image sensing points; 

operation means for operating the three-dimensional 
shape information, on the basis of image data repre- 
senting images obtained with said image sensing means 
and position data representing positions of said image 
sensing means at which the images have been sensed 
by said position detection means; 

conversion means for converting the images of the subject 
sensed at the plurality of image sensing points into an 
image of the subject seen from an arbitrary viewpoint, 
on the basis of the three-dimensional shape information 
operated by said operation means and forming an 
image data file based on a computer file format com- 
prised of the converted image; and 

linking means for linking the image data file of the 
converted image with another data file. 
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